Nov 02 21:33:30.715039 ip-10-0-128-36 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:33:31.084712 ip-10-0-128-36 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:33:31.084712 ip-10-0-128-36 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:33:31.084712 ip-10-0-128-36 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:33:31.084712 ip-10-0-128-36 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:33:31.087835 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087758 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.087835 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087833 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.087835 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087837 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087841 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087844 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087847 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087850 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087853 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087856 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087858 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087861 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087864 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087867 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087870 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087872 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087875 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087878 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087880 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087883 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087886 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087888 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087891 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.087930 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087894 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087896 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087899 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087908 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087911 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087913 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087916 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087918 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087921 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087924 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087926 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087929 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087932 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087934 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087938 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087940 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087946 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087950 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087952 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087955 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.088388 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087958 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087960 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087963 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087966 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087968 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087971 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087973 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087976 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087979 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087981 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087984 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087986 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087989 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087991 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087996 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.087999 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088003 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088006 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088009 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.088865 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088012 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088014 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088017 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088019 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088022 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088024 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088027 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088029 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088032 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088034 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088037 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088039 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088042 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088044 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088047 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088049 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088052 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088054 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088057 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.089312 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088059 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088062 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088064 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088068 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088071 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088073 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088076 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088078 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088083 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088085 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088088 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088090 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088093 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088095 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088098 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088100 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088103 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088105 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088108 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088110 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.089784 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088113 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088116 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088118 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088121 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088123 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088127 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088896 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088905 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088908 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088911 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088914 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088917 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088919 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088923 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088926 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088928 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088931 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088934 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088937 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.090295 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088941 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088946 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088949 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088952 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088954 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088957 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088959 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088962 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088965 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088967 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088970 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088972 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088975 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088977 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088980 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088982 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088985 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088988 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088991 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088994 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.090774 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088997 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.088999 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089002 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089005 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089007 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089010 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089013 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089015 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089018 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089020 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089023 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089025 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089028 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089031 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089033 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089036 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089039 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089041 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089044 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089047 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.091249 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089049 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089053 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089055 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089058 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089060 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089063 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089066 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089068 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089071 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089073 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089076 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089078 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089081 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089084 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089086 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089089 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089091 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089094 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089096 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089099 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.091728 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089101 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089103 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089106 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089108 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089111 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089114 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089116 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089119 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089123 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089127 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089130 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089133 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089135 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089138 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089141 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089143 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089146 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089149 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089151 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089154 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.092201 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089156 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089159 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089162 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089164 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089168 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089171 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089173 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089176 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089178 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089181 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089184 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089186 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089189 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089682 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089688 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089692 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089695 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089701 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089704 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089707 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.092663 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089709 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089712 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089715 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089718 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089721 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089723 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089726 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089728 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089730 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089733 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089735 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089738 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089742 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089745 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089748 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089750 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089753 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089756 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089759 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089761 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.093151 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089764 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089766 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089769 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089772 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089774 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089777 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089779 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089782 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089784 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089788 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089791 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089793 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089796 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089799 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089801 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089804 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089807 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089810 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089812 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089815 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.093616 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089817 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089820 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089824 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089827 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089830 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089833 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089836 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089838 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089841 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089843 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089846 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089849 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089851 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089854 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089856 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089858 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089861 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089863 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089866 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.094116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089868 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089871 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089874 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089876 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089879 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089881 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089883 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089886 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089888 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089891 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089893 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089895 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089898 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089900 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089903 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089906 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089910 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089913 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089916 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089918 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.094559 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089921 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089923 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089926 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089928 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089931 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089933 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089936 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089938 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089941 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089943 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089946 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089948 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089950 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089953 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089956 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089958 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089961 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089965 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089967 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.089970 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.095128 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090755 2582 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090766 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090777 2582 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090783 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090787 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090791 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090795 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090799 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090802 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090821 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090826 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090829 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090833 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090836 2582 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090839 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090842 2582 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090845 2582 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090852 2582 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090856 2582 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090859 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090863 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090866 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090869 2582 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090873 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:33:31.095590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090876 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090879 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090883 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090886 2582 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090889 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090892 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090896 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090898 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090903 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090906 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090908 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090911 2582 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090914 2582 flags.go:64] FLAG: --enable-server="true" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090917 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090921 2582 flags.go:64] FLAG: --event-burst="100" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090924 2582 flags.go:64] FLAG: --event-qps="50" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090928 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090931 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090935 2582 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090939 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090942 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090945 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090948 2582 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090951 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090954 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:33:31.096181 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090957 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090961 2582 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090964 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090967 2582 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090970 2582 flags.go:64] FLAG: --feature-gates="" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090974 2582 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090977 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090980 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090983 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090986 2582 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090990 2582 flags.go:64] FLAG: --help="false" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090993 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090996 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.090999 2582 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091002 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091005 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091008 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091011 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091014 2582 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091017 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091020 2582 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091023 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091026 2582 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091029 2582 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:33:31.096837 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091036 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091039 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091042 2582 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091045 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091048 2582 flags.go:64] FLAG: --lock-file="" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091051 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091054 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091057 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091062 2582 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091066 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091069 2582 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091072 2582 flags.go:64] FLAG: --logging-format="text" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091075 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091078 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091081 2582 flags.go:64] FLAG: --manifest-url="" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091084 2582 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091088 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091093 2582 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091097 2582 flags.go:64] FLAG: --max-pods="110" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091100 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091103 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091106 2582 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091109 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091112 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091115 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:33:31.097407 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091118 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091124 2582 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091127 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091130 2582 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091133 2582 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091137 2582 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091140 2582 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091144 2582 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091147 2582 flags.go:64] FLAG: --port="10250" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091150 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091153 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0cff16335a8165556" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091156 2582 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091159 2582 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091162 2582 flags.go:64] FLAG: --register-node="true" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091165 2582 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091169 2582 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091172 2582 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091175 2582 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091178 2582 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091182 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091185 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091188 2582 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091191 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091194 2582 flags.go:64] FLAG: --runonce="false" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091198 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091201 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091204 2582 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:33:31.098011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091207 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091210 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091213 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091216 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091219 2582 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091222 2582 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091225 2582 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091228 2582 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091230 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091233 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091236 2582 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091239 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091246 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091249 2582 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091251 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091256 2582 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091259 2582 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091262 2582 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091265 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091268 2582 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091271 2582 flags.go:64] FLAG: --v="2" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091275 2582 flags.go:64] FLAG: --version="false" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091279 2582 flags.go:64] FLAG: --vmodule="" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091283 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091287 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:33:31.098755 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091420 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091424 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091427 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091430 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091434 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091437 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091440 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091442 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091445 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091447 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091450 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091453 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091456 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091459 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091461 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091464 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091466 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091469 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091471 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091475 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.099345 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091478 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091481 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091484 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091486 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091489 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091491 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091494 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091497 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091500 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091502 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091505 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091507 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091510 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091513 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091515 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091518 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091522 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091524 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091529 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.099913 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091532 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091535 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091537 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091540 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091543 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091546 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091550 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091553 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091556 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091558 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091561 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091563 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091567 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091569 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091572 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091574 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091577 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091579 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091582 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.100370 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091584 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091587 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091590 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091592 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091595 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091597 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091600 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091603 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091605 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091608 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091612 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091614 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091617 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091619 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091622 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091624 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091627 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091630 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091632 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091634 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.100834 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091637 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091639 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091642 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091644 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091647 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091650 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091653 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091656 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091658 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091661 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091663 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091666 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091680 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091682 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091685 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091688 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091690 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091693 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091695 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091698 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.101304 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091701 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091704 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091708 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091711 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091714 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091716 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091719 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091721 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.091747 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091788 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091794 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091797 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091800 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091803 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091805 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091808 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.101797 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091810 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091815 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091817 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091820 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091823 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091826 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091830 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091834 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091837 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091840 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091843 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091847 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091851 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091854 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091856 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091859 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091862 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091865 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.102175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091867 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091872 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091874 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091877 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091879 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091882 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091884 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091887 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091890 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091892 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091895 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091897 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091899 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091902 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091905 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091908 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091911 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091914 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091916 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.102601 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091919 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091921 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091924 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091927 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091929 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091932 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091934 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091937 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091939 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091942 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091945 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091947 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091950 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091953 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091957 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091959 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091962 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091964 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091967 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091970 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.103137 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091972 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091975 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091977 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091980 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091982 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091985 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091987 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091990 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091994 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091997 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.091999 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092002 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092004 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092007 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092009 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092012 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092014 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092017 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092020 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092022 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.103607 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092025 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092028 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092030 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092033 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092035 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092038 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092042 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092045 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092047 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092050 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092052 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092055 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092057 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092060 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092062 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092065 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092067 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092071 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092074 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092076 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.104116 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092080 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.092083 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.099369 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.099386 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.099405 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.099410 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099480 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099486 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099490 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099494 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099497 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099500 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099503 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099505 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099508 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099511 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099513 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099516 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099518 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.104590 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099521 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099524 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099526 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099529 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099532 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099536 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099540 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099545 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099549 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099552 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099555 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099558 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099560 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099563 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099566 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099569 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099578 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099581 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099584 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099586 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.105046 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099589 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099591 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099594 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099596 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099599 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099601 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099604 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099607 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099609 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099613 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099617 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099622 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099626 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099630 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099634 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099637 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099640 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099643 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099645 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.105524 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099648 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099650 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099653 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099656 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099658 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099661 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099664 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099666 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099683 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099688 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099700 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099703 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099706 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099709 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099711 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099714 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099717 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099720 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099723 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099725 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.106081 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099728 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099730 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099733 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099735 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099738 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099741 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099743 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099746 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099749 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099766 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099770 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099772 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099775 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099777 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099780 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099782 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099785 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099787 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099790 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099792 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.106896 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099795 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099797 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099800 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099808 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099811 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099814 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099816 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099818 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099821 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099823 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099826 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099830 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099834 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.099839 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.099868 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:33:31.107554 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100081 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100087 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100089 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100092 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100095 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100097 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100100 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100102 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100105 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100107 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100110 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100112 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100115 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100118 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100120 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100123 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100125 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100128 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100130 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100134 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.107981 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100138 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100151 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100157 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100160 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100163 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100165 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100168 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100170 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100173 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100175 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100178 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100181 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100183 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100186 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100189 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100191 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100194 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100196 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100199 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.108442 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100201 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100203 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100206 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100208 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100211 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100213 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100216 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100219 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100222 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100226 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100231 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100235 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100238 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100241 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100243 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100254 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100257 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100259 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100262 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.108901 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100265 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100267 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100270 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100272 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100275 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100277 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100280 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100284 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100287 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100290 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100292 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100295 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100298 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100303 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100307 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100311 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100314 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100317 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100320 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.109340 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100322 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100325 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100327 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100330 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100332 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100335 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100337 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100340 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100342 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100345 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100353 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100356 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100359 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100361 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100364 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100366 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100369 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100373 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100377 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100382 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.109811 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100388 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100392 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100395 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100397 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100400 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100402 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100405 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100408 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100411 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.100434 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100480 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100485 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100487 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100491 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100493 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:31.110270 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100496 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100498 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100501 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100503 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100506 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100508 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100511 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100513 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100522 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100525 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100527 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100531 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100534 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100536 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100539 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100542 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100544 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100549 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100551 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100555 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:31.110622 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100559 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100563 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100567 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100571 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100575 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100578 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100581 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100584 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100586 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100589 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100591 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100594 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100596 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100599 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100601 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100604 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100606 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100609 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100612 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:31.111328 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100614 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100616 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100625 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100628 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100632 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100634 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100637 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100639 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100642 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100644 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100649 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100652 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100654 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100657 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100659 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100663 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100680 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100685 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100689 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:31.111857 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100692 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100695 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100698 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100701 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100704 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100706 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100709 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100712 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100714 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100717 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100719 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100722 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100724 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100727 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100729 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100732 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100741 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100745 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100748 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100750 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:31.112316 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100753 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100755 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100758 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100761 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100764 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100766 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100769 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100772 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100774 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100783 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100785 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100788 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100791 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100793 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100797 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100799 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100802 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100804 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100807 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:31.112843 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100809 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100812 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100814 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:31.100817 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.100877 2582 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.102899 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.103688 2582 server.go:1014] "Starting client certificate rotation" Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.103781 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.104371 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.107270 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:33:31.113280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.110704 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:33:31.119878 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.119862 2582 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:33:31.122692 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.122680 2582 log.go:25] "Validated CRI v1 image API" Nov 02 21:33:31.123466 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.123442 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:33:31.126983 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.126954 2582 fs.go:135] Filesystem UUIDs: map[49e0b128-9751-4971-9fa9-1264b3343913:/dev/nvme0n1p3 58667e7a-9569-4cbd-88b9-b12113450898:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 21:33:31.127076 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.126979 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:33:31.132027 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.131910 2582 manager.go:217] Machine: {Timestamp:2026-11-02 21:33:31.130609634 +0000 UTC m=+0.302982275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099438 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b46141ec9da8951268bb0bf599ec6 SystemUUID:ec2b4614-1ec9-da89-5126-8bb0bf599ec6 BootID:43bd0896-3e71-4a74-8bce-20186123f69f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:69:a0:4d:f2:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:69:a0:4d:f2:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:e4:ad:cb:40:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:33:31.132625 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.132613 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:33:31.132779 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.132763 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:33:31.134472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.134448 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:33:31.134600 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.134473 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-36.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:33:31.134662 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.134602 2582 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:33:31.134662 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.134610 2582 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:33:31.134662 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.134657 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:33:31.134748 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.134688 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:33:31.135990 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.135980 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:33:31.140143 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140131 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:33:31.140255 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140245 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:33:31.140887 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140877 2582 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:33:31.140922 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140893 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:33:31.140922 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140903 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:33:31.140922 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140911 2582 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:33:31.140922 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.140922 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:33:31.141343 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.141327 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:33:31.144638 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.144610 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:33:31.148881 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.148868 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:33:31.149610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149598 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:33:31.149655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149627 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:33:31.149805 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149792 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:33:31.149835 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149814 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:33:31.149835 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149820 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:33:31.149835 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149827 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:33:31.149835 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149832 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:33:31.149933 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.149837 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:33:31.150292 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.150283 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:33:31.150320 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.150292 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:33:31.150320 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.150311 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:33:31.150320 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.150317 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:33:31.150398 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.150329 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:33:31.150398 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.150338 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:33:31.152475 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.152465 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:33:31.152509 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.152478 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:33:31.157772 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.157755 2582 server.go:1263] "Started kubelet" Nov 02 21:33:31.157886 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.157836 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:33:31.157966 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.157918 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:33:31.158020 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.157986 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:33:31.158446 ip-10-0-128-36 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:33:31.159158 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.159141 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:33:31.160784 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.160770 2582 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.163554 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.164159 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.164275 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.164882 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.164904 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.164918 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.165074 2582 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.165081 2582 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.165227 2582 factory.go:55] Registering systemd factory Nov 02 21:33:31.165621 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.165240 2582 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:33:31.166111 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.166097 2582 factory.go:153] Registering CRI-O factory Nov 02 21:33:31.166163 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.166115 2582 factory.go:223] Registration of the crio container factory successfully Nov 02 21:33:31.166209 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.166190 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:33:31.166254 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.166220 2582 factory.go:103] Registering Raw factory Nov 02 21:33:31.166254 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.166236 2582 manager.go:1199] Started watching for new ooms in manager Nov 02 21:33:31.166422 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.166403 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:33:31.167101 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.167085 2582 manager.go:319] Starting recovery of all containers Nov 02 21:33:31.176384 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.176367 2582 manager.go:324] Recovery completed Nov 02 21:33:31.182598 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.182579 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.184901 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.184875 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.184991 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.184917 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.184991 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.184931 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.185430 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.185415 2582 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:33:31.185479 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.185429 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:33:31.185479 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.185445 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:33:31.185479 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.185429 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.186202 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.186186 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.186275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.186210 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.186275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.186225 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.187531 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.187513 2582 policy_none.go:50] "Start" Nov 02 21:33:31.187600 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.187535 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:33:31.187600 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.187552 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:33:31.189592 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.189577 2582 policy_none.go:44] "Start" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.221631 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.226938 2582 manager.go:341] "Starting Device Plugin manager" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.226965 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.226975 2582 server.go:87] "Starting device plugin registration server" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.227207 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.227218 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.227387 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.227473 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.227482 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.228047 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.228093 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.232113 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.233182 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.233208 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.233232 2582 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:33:31.243227 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.233270 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:33:31.302895 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.302854 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-36.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:33:31.303021 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.302906 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-36.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:33:31.303854 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.302842 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-36.us-west-1.compute.internal.18e457e94e959994 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-36.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-36.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 21:33:31.15772354 +0000 UTC m=+0.330096179,LastTimestamp:2026-11-02 21:33:31.15772354 +0000 UTC m=+0.330096179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-36.us-west-1.compute.internal,}" Nov 02 21:33:31.327435 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.327415 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.328231 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.328217 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.328304 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.328249 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.328304 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.328259 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.328304 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.328282 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.334623 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.334608 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal"] Nov 02 21:33:31.334690 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.334679 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.335417 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.335359 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.335417 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.335389 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.335417 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.335402 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.337459 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.337447 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.337609 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.337594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.337694 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.337622 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.338163 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.338149 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.338163 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.338157 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.338264 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.338176 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.338264 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.338178 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.338264 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.338186 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.338264 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.338195 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.340178 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.340163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.340244 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.340184 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.340885 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.340872 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.340947 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.340896 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.340947 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.340905 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.366062 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.366039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.366139 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.366064 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.366139 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.366079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356429dd5260af9a132a468067041120-config\") pod \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"356429dd5260af9a132a468067041120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.374896 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.374870 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.377377 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.377359 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.377625 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.377611 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:33:31.377692 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.377661 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r4nx8" Nov 02 21:33:31.387552 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.387529 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-36.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 21:33:31.387552 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.387532 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 21:33:31.387659 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.387569 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 21:33:31.387659 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.387516 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-36.us-west-1.compute.internal.18e457e950344ddc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-36.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-36.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-36.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 21:33:31.184901596 +0000 UTC m=+0.357274236,LastTimestamp:2026-11-02 21:33:31.184901596 +0000 UTC m=+0.357274236,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-36.us-west-1.compute.internal,}" Nov 02 21:33:31.387843 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.387822 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 21:33:31.458977 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.458956 2582 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-128-36.us-west-1.compute.internal\" not found" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.466353 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.466332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.466441 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.466361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.466441 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.466389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356429dd5260af9a132a468067041120-config\") pod \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"356429dd5260af9a132a468067041120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.466532 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.466449 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356429dd5260af9a132a468067041120-config\") pod \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"356429dd5260af9a132a468067041120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.466532 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.466453 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.466532 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.466453 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.604869 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.604797 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-36.us-west-1.compute.internal\" not found" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.659971 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.659941 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:31.660875 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.660855 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:31.660957 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.660888 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:31.660957 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.660905 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:31.660957 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.660937 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.678007 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.677990 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.682113 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.682095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.717575 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:31.717429 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:31.717683 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.717581 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-36.us-west-1.compute.internal\": node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:31.835235 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.835207 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:31.936347 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:31.936279 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:32.036361 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:32.036331 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:32.103768 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.103737 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:33:32.104297 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:32.104165 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a844a01bd9c1a45928f36aeb9963b228-b8a9b0af8073bf35.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-36.us-west-1.compute.internal.18e457e95034a787\": read tcp 10.0.128.36:35274->3.130.5.189:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-36.us-west-1.compute.internal.18e457e95034a787 default 4816 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-36.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-36.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-36.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 21:33:31 +0000 UTC,LastTimestamp:2026-11-02 21:33:31.335395174 +0000 UTC m=+0.507767815,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-36.us-west-1.compute.internal,}" Nov 02 21:33:32.137007 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:32.136988 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:32.237985 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:32.237933 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:32.340332 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:32.340306 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:32.370834 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.370816 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:32.371814 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.371796 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:32.371918 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.371833 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:32.371918 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.371849 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:32.383555 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.383529 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:28:31 +0000 UTC" deadline="2028-07-14 18:59:13.39195925 +0000 UTC" Nov 02 21:33:32.383555 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.383555 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14877h25m41.008408413s" Nov 02 21:33:32.441491 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:32.441468 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:33:32.472077 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.472055 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:32.524179 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.524121 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:32.565085 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.565063 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:32.631909 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.631891 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:33:32.633829 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.633815 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:33:32.721942 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.721923 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:33:32.722098 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.721986 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:32.757556 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.757527 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:32.865133 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:32.865100 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356429dd5260af9a132a468067041120.slice/crio-e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e WatchSource:0}: Error finding container e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e: Status 404 returned error can't find the container with id e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e Nov 02 21:33:32.865579 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:32.865553 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6ec7fa97b5465128bc132a816f76dd.slice/crio-5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d WatchSource:0}: Error finding container 5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d: Status 404 returned error can't find the container with id 5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d Nov 02 21:33:32.870489 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:32.870473 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:33:33.143040 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.142974 2582 apiserver.go:52] "Watching apiserver" Nov 02 21:33:33.166103 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.166074 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:33:33.237683 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.237635 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"356429dd5260af9a132a468067041120","Type":"ContainerStarted","Data":"e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e"} Nov 02 21:33:33.238749 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.238725 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"9d6ec7fa97b5465128bc132a816f76dd","Type":"ContainerStarted","Data":"5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d"} Nov 02 21:33:33.253590 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.253570 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:33:33.253919 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.253902 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9xtbf","openshift-network-operator/iptables-alerter-2nwsl","openshift-ovn-kubernetes/ovnkube-node-cswj4","openshift-cluster-node-tuning-operator/tuned-jxnx6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal","openshift-multus/multus-cbj8m","kube-system/global-pull-secret-syncer-sr2gb","kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j","openshift-dns/node-resolver-4qplz","openshift-image-registry/node-ca-9njpz","openshift-multus/multus-additional-cni-plugins-w9h49","openshift-multus/network-metrics-daemon-tfzjr","kube-system/konnectivity-agent-wk8sg"] Nov 02 21:33:33.257919 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.257900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:33.258007 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.257980 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:33.260143 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.260127 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.262562 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.262544 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.264830 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.264813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.267220 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.267187 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.269460 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.269439 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.269562 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.269504 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:33.271864 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.271848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.273882 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.273861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-etc-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.273963 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.273886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz8v\" (UniqueName: \"kubernetes.io/projected/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-kube-api-access-4vz8v\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.273963 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.273922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6dc\" (UniqueName: \"kubernetes.io/projected/d020426a-2902-454c-8b86-7ecc7c666826-kube-api-access-qm6dc\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.273963 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.273958 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.274063 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.273993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-daemon-config\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274100 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274062 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-var-lib-kubelet\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274234 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274097 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78t6\" (UniqueName: \"kubernetes.io/projected/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-kube-api-access-n78t6\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274234 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.274234 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-run\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274380 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cni-binary-copy\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274380 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-tmp\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274380 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-k8s-cni-cncf-io\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274380 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-slash\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.274547 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-log-socket\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.274547 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysconfig\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274547 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-kubernetes\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274547 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-kubelet\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274708 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovn-node-metrics-cert\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.274708 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-conf\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274708 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274680 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-tuned\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.274860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274721 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-netns\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-multus\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-conf-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.274860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dgc\" (UniqueName: \"kubernetes.io/projected/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-kube-api-access-c5dgc\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-systemd-units\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-ovn\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-lib-modules\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.275011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274962 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d020426a-2902-454c-8b86-7ecc7c666826-host-slash\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.275011 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.274985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-modprobe-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.275224 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-hostroot\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275224 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-multus-certs\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275224 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:33.275224 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275165 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-bin\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275224 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-config\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-etc-kubernetes\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-env-overrides\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-kubelet\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-node-log\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-systemd\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.275416 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-socket-dir-parent\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-netns\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-script-lib\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275499 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-sys\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cnibin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-bin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-netd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d020426a-2902-454c-8b86-7ecc7c666826-iptables-alerter-script\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-systemd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-var-lib-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-host\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.275801 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.276194 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.275834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-system-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.276465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.276441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.278825 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.278808 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.282917 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.281591 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.284274 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.284257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.284346 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.284316 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:33.286598 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.286576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.315909 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.315888 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.316001 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.315979 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.316057 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.316006 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:33:33.316098 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.316080 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nr5z4\"" Nov 02 21:33:33.318290 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318267 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:33:33.318290 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318283 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:33:33.318451 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318336 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsxz6\"" Nov 02 21:33:33.318451 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318351 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.318451 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318394 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.318596 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318485 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:33:33.318639 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.318620 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:33:33.320008 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.319989 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.320058 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.320011 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95jdp\"" Nov 02 21:33:33.320058 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.320019 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.322284 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.322269 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.322332 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.322292 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7clxp\"" Nov 02 21:33:33.322642 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.322627 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:33:33.322678 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.322656 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:33:33.329765 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.329731 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f7c29\"" Nov 02 21:33:33.329765 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.329750 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:33:33.362777 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.362758 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.362979 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.362903 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.362979 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.362940 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.363106 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.362983 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.363106 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.363007 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2mch2\"" Nov 02 21:33:33.363106 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.363016 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.365666 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.365652 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:33:33.367234 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.367217 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:33:33.367307 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.367272 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:33:33.367307 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.367274 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qbt7b\"" Nov 02 21:33:33.367390 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.367306 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.369054 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369041 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:33:33.369606 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369595 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b8lsz\"" Nov 02 21:33:33.369821 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369787 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sb4td\"" Nov 02 21:33:33.369924 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369835 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:33:33.369924 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369837 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:33:33.369924 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369846 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.369924 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369869 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8wpx" Nov 02 21:33:33.369924 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.369847 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:33:33.376497 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-dbus\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.376585 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376502 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cnibin\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.376585 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.376585 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-run\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.376585 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cni-binary-copy\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.376585 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-tmp\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-k8s-cni-cncf-io\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376607 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-sys-fs\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376641 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-k8s-cni-cncf-io\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376717 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-run\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-slash\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.376797 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376759 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-slash\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377096 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.376933 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:33:33.377096 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377071 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cni-binary-copy\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377315 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-log-socket\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377356 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysconfig\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377396 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-kubernetes\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377396 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377369 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-log-socket\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-kubelet\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377398 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysconfig\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-kubelet\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377437 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-kubernetes\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-socket-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovn-node-metrics-cert\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-conf\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-tuned\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-netns\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-multus\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.377620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-system-cni-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-conf-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dgc\" (UniqueName: \"kubernetes.io/projected/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-kube-api-access-c5dgc\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-systemd-units\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377722 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-conf-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-ovn\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377691 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-multus\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377766 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-ovn\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377779 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-conf\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-lib-modules\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-kubelet-config\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377837 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-systemd-units\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377867 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-netns\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d020426a-2902-454c-8b86-7ecc7c666826-host-slash\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-modprobe-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.377970 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.377923 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d020426a-2902-454c-8b86-7ecc7c666826-host-slash\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-hostroot\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378027 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-modprobe-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378082 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-hostroot\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378095 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-lib-modules\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-multus-certs\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378134 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-multus-certs\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb533acd-73cb-4f30-a60e-e96debb69f25-konnectivity-ca\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7rx\" (UniqueName: \"kubernetes.io/projected/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-kube-api-access-fw7rx\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-bin\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-config\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378343 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-bin\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-registration-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6pr\" (UniqueName: \"kubernetes.io/projected/0d0a36a5-f22c-4759-8e09-724db23b7c04-kube-api-access-8s6pr\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.378758 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378452 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-etc-kubernetes\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378479 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-etc-kubernetes\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-env-overrides\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-kubelet\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-node-log\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-systemd\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-socket-dir-parent\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378631 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-kubelet\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-netns\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-script-lib\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378717 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-systemd\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-sys\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cnibin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378775 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-sys\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-bin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378813 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-node-log\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.379523 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378851 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-config\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-device-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378899 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-socket-dir-parent\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378923 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-env-overrides\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378941 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378962 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-bin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqts\" (UniqueName: \"kubernetes.io/projected/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kube-api-access-mlqts\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378976 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-netns\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.378978 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cnibin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-netd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379123 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-netd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb533acd-73cb-4f30-a60e-e96debb69f25-agent-certs\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d020426a-2902-454c-8b86-7ecc7c666826-iptables-alerter-script\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-systemd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-var-lib-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380239 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379376 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-systemd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-host\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-hosts-file\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379433 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-script-lib\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d0a36a5-f22c-4759-8e09-724db23b7c04-host\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379438 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-var-lib-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8kt\" (UniqueName: \"kubernetes.io/projected/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-kube-api-access-rp8kt\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379486 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-host\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-system-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-tmp-dir\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-etc-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379670 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d020426a-2902-454c-8b86-7ecc7c666826-iptables-alerter-script\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz8v\" (UniqueName: \"kubernetes.io/projected/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-kube-api-access-4vz8v\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6dc\" (UniqueName: \"kubernetes.io/projected/d020426a-2902-454c-8b86-7ecc7c666826-kube-api-access-qm6dc\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.380764 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379810 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-system-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-daemon-config\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379842 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379872 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-etc-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379894 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnd2\" (UniqueName: \"kubernetes.io/projected/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-kube-api-access-rnnd2\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.379970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d0a36a5-f22c-4759-8e09-724db23b7c04-serviceca\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.380002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-var-lib-kubelet\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.380050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78t6\" (UniqueName: \"kubernetes.io/projected/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-kube-api-access-n78t6\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.380071 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-var-lib-kubelet\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.380313 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-daemon-config\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.381494 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovn-node-metrics-cert\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.381551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.381516 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-tuned\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.382008 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.381481 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-tmp\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.437156 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.437108 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:33.437156 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.437135 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:33.437156 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.437150 2582 projected.go:196] Error preparing data for projected volume kube-api-access-bffx9 for pod openshift-network-diagnostics/network-check-target-9xtbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:33.437306 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.437215 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9 podName:e987d331-5974-430e-8a6b-d1b1dcbb7fe9 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:33.9371957 +0000 UTC m=+3.109568344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bffx9" (UniqueName: "kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9") pod "network-check-target-9xtbf" (UID: "e987d331-5974-430e-8a6b-d1b1dcbb7fe9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:33.439415 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.439388 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dgc\" (UniqueName: \"kubernetes.io/projected/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-kube-api-access-c5dgc\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.439861 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.439834 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6dc\" (UniqueName: \"kubernetes.io/projected/d020426a-2902-454c-8b86-7ecc7c666826-kube-api-access-qm6dc\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.439930 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.439912 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz8v\" (UniqueName: \"kubernetes.io/projected/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-kube-api-access-4vz8v\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.440899 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.440881 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78t6\" (UniqueName: \"kubernetes.io/projected/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-kube-api-access-n78t6\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.480477 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-registration-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.480569 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6pr\" (UniqueName: \"kubernetes.io/projected/0d0a36a5-f22c-4759-8e09-724db23b7c04-kube-api-access-8s6pr\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.480569 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.480569 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-device-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.480569 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqts\" (UniqueName: \"kubernetes.io/projected/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kube-api-access-mlqts\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.480569 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480568 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-device-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480583 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-registration-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.480639 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb533acd-73cb-4f30-a60e-e96debb69f25-agent-certs\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-hosts-file\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d0a36a5-f22c-4759-8e09-724db23b7c04-host\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.480705 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.480752 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:33:33.980731473 +0000 UTC m=+3.153104113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480752 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-hosts-file\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480751 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d0a36a5-f22c-4759-8e09-724db23b7c04-host\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480777 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8kt\" (UniqueName: \"kubernetes.io/projected/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-kube-api-access-rp8kt\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.480789 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret podName:4016e9e3-e3e8-454a-a635-49970015c350 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:33.980771675 +0000 UTC m=+3.153144306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret") pod "global-pull-secret-syncer-sr2gb" (UID: "4016e9e3-e3e8-454a-a635-49970015c350") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-tmp-dir\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.480868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnd2\" (UniqueName: \"kubernetes.io/projected/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-kube-api-access-rnnd2\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d0a36a5-f22c-4759-8e09-724db23b7c04-serviceca\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480956 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.480980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-dbus\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cnibin\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-sys-fs\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481071 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cnibin\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481077 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-dbus\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481118 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-sys-fs\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-socket-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481141 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-tmp-dir\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-system-cni-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.481656 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-kubelet-config\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb533acd-73cb-4f30-a60e-e96debb69f25-konnectivity-ca\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481280 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481284 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-socket-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481290 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7rx\" (UniqueName: \"kubernetes.io/projected/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-kube-api-access-fw7rx\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481338 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-kubelet-config\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481350 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-system-cni-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481554 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481669 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481801 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb533acd-73cb-4f30-a60e-e96debb69f25-konnectivity-ca\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.481860 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.482164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.482005 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d0a36a5-f22c-4759-8e09-724db23b7c04-serviceca\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.482794 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.482776 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb533acd-73cb-4f30-a60e-e96debb69f25-agent-certs\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.541484 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.541444 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6pr\" (UniqueName: \"kubernetes.io/projected/0d0a36a5-f22c-4759-8e09-724db23b7c04-kube-api-access-8s6pr\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.541948 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.541920 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqts\" (UniqueName: \"kubernetes.io/projected/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kube-api-access-mlqts\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.541948 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.541918 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7rx\" (UniqueName: \"kubernetes.io/projected/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-kube-api-access-fw7rx\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.542166 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.542139 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnd2\" (UniqueName: \"kubernetes.io/projected/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-kube-api-access-rnnd2\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.542209 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.542186 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8kt\" (UniqueName: \"kubernetes.io/projected/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-kube-api-access-rp8kt\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.573130 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.573110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:33:33.583971 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.583949 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd020426a_2902_454c_8b86_7ecc7c666826.slice/crio-16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db WatchSource:0}: Error finding container 16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db: Status 404 returned error can't find the container with id 16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db Nov 02 21:33:33.595401 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.595384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:33:33.601477 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.601455 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:33.601580 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.601502 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbj8m" Nov 02 21:33:33.603352 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.603333 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc532d99_0b1a_40fb_ad7c_42c4cfbb879e.slice/crio-ff64458cd117cfe1ad465d0fad9b0c50c283e665c1eea7d88636cbc25e6d5c25 WatchSource:0}: Error finding container ff64458cd117cfe1ad465d0fad9b0c50c283e665c1eea7d88636cbc25e6d5c25: Status 404 returned error can't find the container with id ff64458cd117cfe1ad465d0fad9b0c50c283e665c1eea7d88636cbc25e6d5c25 Nov 02 21:33:33.608682 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.608661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:33:33.610335 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.610314 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6466fd_6cf7_4d27_bd08_4eaf51c5990e.slice/crio-6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27 WatchSource:0}: Error finding container 6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27: Status 404 returned error can't find the container with id 6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27 Nov 02 21:33:33.610961 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.610939 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4qplz" Nov 02 21:33:33.613196 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.613179 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:33:33.614247 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.614227 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1c4210_57de_4ca9_aad7_3fc8a8b8ff0c.slice/crio-324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f WatchSource:0}: Error finding container 324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f: Status 404 returned error can't find the container with id 324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f Nov 02 21:33:33.618718 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.618700 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe41410a_c20e_48c0_b3c3_f531c79c3fe5.slice/crio-b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049 WatchSource:0}: Error finding container b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049: Status 404 returned error can't find the container with id b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049 Nov 02 21:33:33.623678 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.623656 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557fdf7f_9d7a_4de7_9819_becbc1fbfb8a.slice/crio-4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df WatchSource:0}: Error finding container 4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df: Status 404 returned error can't find the container with id 4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df Nov 02 21:33:33.625718 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.625697 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0a36a5_f22c_4759_8e09_724db23b7c04.slice/crio-d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4 WatchSource:0}: Error finding container d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4: Status 404 returned error can't find the container with id d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4 Nov 02 21:33:33.628861 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.628845 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:33.633772 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.633759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:33:33.637966 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.637947 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb533acd_73cb_4f30_a60e_e96debb69f25.slice/crio-8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06 WatchSource:0}: Error finding container 8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06: Status 404 returned error can't find the container with id 8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06 Nov 02 21:33:33.642949 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:33:33.642929 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d0b0db_ceaa_49c5_bff5_16b1d7c370e3.slice/crio-3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5 WatchSource:0}: Error finding container 3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5: Status 404 returned error can't find the container with id 3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5 Nov 02 21:33:33.985633 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.985598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:33.985842 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.985657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:33.985842 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:33.985707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:33.985842 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.985831 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:33.986027 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.985894 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:33:34.985873113 +0000 UTC m=+4.158245747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:33.986027 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.985987 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:33.986027 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.986002 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:33.986027 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.986014 2582 projected.go:196] Error preparing data for projected volume kube-api-access-bffx9 for pod openshift-network-diagnostics/network-check-target-9xtbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:33.986236 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.986047 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9 podName:e987d331-5974-430e-8a6b-d1b1dcbb7fe9 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:34.986038351 +0000 UTC m=+4.158410980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bffx9" (UniqueName: "kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9") pod "network-check-target-9xtbf" (UID: "e987d331-5974-430e-8a6b-d1b1dcbb7fe9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:33.986236 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.986108 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:33.986236 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:33.986134 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret podName:4016e9e3-e3e8-454a-a635-49970015c350 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:34.986126658 +0000 UTC m=+4.158499287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret") pod "global-pull-secret-syncer-sr2gb" (UID: "4016e9e3-e3e8-454a-a635-49970015c350") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:34.242863 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.242771 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27"} Nov 02 21:33:34.244255 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.244222 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2nwsl" event={"ID":"d020426a-2902-454c-8b86-7ecc7c666826","Type":"ContainerStarted","Data":"16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db"} Nov 02 21:33:34.246043 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.246010 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9njpz" event={"ID":"0d0a36a5-f22c-4759-8e09-724db23b7c04","Type":"ContainerStarted","Data":"d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4"} Nov 02 21:33:34.247438 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.247404 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" event={"ID":"fe41410a-c20e-48c0-b3c3-f531c79c3fe5","Type":"ContainerStarted","Data":"b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049"} Nov 02 21:33:34.248892 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.248849 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbj8m" event={"ID":"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c","Type":"ContainerStarted","Data":"324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f"} Nov 02 21:33:34.250268 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.250245 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" event={"ID":"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e","Type":"ContainerStarted","Data":"ff64458cd117cfe1ad465d0fad9b0c50c283e665c1eea7d88636cbc25e6d5c25"} Nov 02 21:33:34.251881 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.251847 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerStarted","Data":"3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5"} Nov 02 21:33:34.253253 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.253222 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wk8sg" event={"ID":"eb533acd-73cb-4f30-a60e-e96debb69f25","Type":"ContainerStarted","Data":"8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06"} Nov 02 21:33:34.254520 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.254493 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4qplz" event={"ID":"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a","Type":"ContainerStarted","Data":"4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df"} Nov 02 21:33:34.373402 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.373338 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:28:33 +0000 UTC" deadline="2028-06-30 17:21:12.647564653 +0000 UTC" Nov 02 21:33:34.373402 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.373391 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14539h47m38.274179165s" Nov 02 21:33:34.993529 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.993486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:34.993732 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.993577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:34.993732 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:34.993622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:34.993931 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.993769 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:34.993931 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.993831 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret podName:4016e9e3-e3e8-454a-a635-49970015c350 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:36.993813329 +0000 UTC m=+6.166185973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret") pod "global-pull-secret-syncer-sr2gb" (UID: "4016e9e3-e3e8-454a-a635-49970015c350") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:34.993931 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.993898 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:34.993931 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.993933 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:33:36.9939218 +0000 UTC m=+6.166294445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:34.994152 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.994019 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:34.994152 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.994034 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:34.994152 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.994047 2582 projected.go:196] Error preparing data for projected volume kube-api-access-bffx9 for pod openshift-network-diagnostics/network-check-target-9xtbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:34.994152 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:34.994084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9 podName:e987d331-5974-430e-8a6b-d1b1dcbb7fe9 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:36.994072277 +0000 UTC m=+6.166444922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bffx9" (UniqueName: "kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9") pod "network-check-target-9xtbf" (UID: "e987d331-5974-430e-8a6b-d1b1dcbb7fe9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:35.238862 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:35.238828 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:35.239037 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:35.238971 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:35.239104 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:35.239063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:35.239165 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:35.239134 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:35.239220 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:35.239199 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:35.239299 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:35.239280 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:35.374627 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:35.374510 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:28:33 +0000 UTC" deadline="2028-06-24 19:21:58.877499337 +0000 UTC" Nov 02 21:33:35.374627 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:35.374576 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14397h48m23.502927849s" Nov 02 21:33:37.010221 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.009866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:37.010221 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.010175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:37.010221 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010199 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:37.010221 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.010214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010232 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010247 2582 projected.go:196] Error preparing data for projected volume kube-api-access-bffx9 for pod openshift-network-diagnostics/network-check-target-9xtbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010311 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9 podName:e987d331-5974-430e-8a6b-d1b1dcbb7fe9 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:41.010290423 +0000 UTC m=+10.182663057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bffx9" (UniqueName: "kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9") pod "network-check-target-9xtbf" (UID: "e987d331-5974-430e-8a6b-d1b1dcbb7fe9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010338 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010387 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010392 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:33:41.010375365 +0000 UTC m=+10.182747998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:37.011035 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.010425 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret podName:4016e9e3-e3e8-454a-a635-49970015c350 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:41.010412884 +0000 UTC m=+10.182785515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret") pod "global-pull-secret-syncer-sr2gb" (UID: "4016e9e3-e3e8-454a-a635-49970015c350") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:37.235800 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.235695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:37.236038 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.235846 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:37.236038 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.235887 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:37.236038 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.235969 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:37.236038 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.236009 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:37.236308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:37.236067 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:37.283099 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.281957 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d6ec7fa97b5465128bc132a816f76dd" containerID="4c35409c103b018a01438eddff6081d0316600f898c6bec7a4a1ac9d67b5af5c" exitCode=0 Nov 02 21:33:37.283099 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.282035 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"9d6ec7fa97b5465128bc132a816f76dd","Type":"ContainerDied","Data":"4c35409c103b018a01438eddff6081d0316600f898c6bec7a4a1ac9d67b5af5c"} Nov 02 21:33:37.301845 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:37.301815 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"356429dd5260af9a132a468067041120","Type":"ContainerStarted","Data":"b9c5923ada059b2985b256c9a23d6ba32fa7854ebf2707e48c654d198e5511da"} Nov 02 21:33:38.314018 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:38.313970 2582 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="dc0f427c24582563d16460de302abecf88bc96966307a92e9700148295e7aaaf" exitCode=0 Nov 02 21:33:38.314471 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:38.314075 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerDied","Data":"dc0f427c24582563d16460de302abecf88bc96966307a92e9700148295e7aaaf"} Nov 02 21:33:38.319513 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:38.319449 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"9d6ec7fa97b5465128bc132a816f76dd","Type":"ContainerStarted","Data":"02ad86fdd1565226d09dbf1f986f96ff801061fd5f2516f7bd5a655704cbbe6c"} Nov 02 21:33:38.533548 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:38.533491 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" podStartSLOduration=6.533477904 podStartE2EDuration="6.533477904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:37.505933492 +0000 UTC m=+6.678306144" watchObservedRunningTime="2026-11-02 21:33:38.533477904 +0000 UTC m=+7.705850553" Nov 02 21:33:39.235928 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.235792 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:39.235928 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:39.235918 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:39.236147 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.235991 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:39.236147 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:39.236099 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:39.236147 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.236131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:39.236294 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:39.236213 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:39.327217 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.327183 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wk8sg" event={"ID":"eb533acd-73cb-4f30-a60e-e96debb69f25","Type":"ContainerStarted","Data":"209ebfad6b68da57538dad2ecdb36438fd533d2d567faf2f94a048cb96f9bc40"} Nov 02 21:33:39.416586 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.416558 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:39.417080 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.417060 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:39.438438 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.438335 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" podStartSLOduration=7.438318111 podStartE2EDuration="7.438318111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:38.594919865 +0000 UTC m=+7.767292523" watchObservedRunningTime="2026-11-02 21:33:39.438318111 +0000 UTC m=+8.610690760" Nov 02 21:33:39.438999 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:39.438969 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wk8sg" podStartSLOduration=2.725304935 podStartE2EDuration="7.438956648s" totalImagesPullingTime="4.713651713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:39.437798441 +0000 UTC m=+8.610171131" watchObservedRunningTime="2026-11-02 21:33:39.438956648 +0000 UTC m=+8.611329300" Nov 02 21:33:40.332005 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:40.331924 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9njpz" event={"ID":"0d0a36a5-f22c-4759-8e09-724db23b7c04","Type":"ContainerStarted","Data":"8d0e006f6de9218d8c3366c5ecb247e7a25e444a7025c037bbc8880910e9e9fe"} Nov 02 21:33:40.336069 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:40.336040 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" event={"ID":"fe41410a-c20e-48c0-b3c3-f531c79c3fe5","Type":"ContainerStarted","Data":"0dd7aa8d7e94598a74e08685641c70289cddaae777ba35fa23656f043ba21fe1"} Nov 02 21:33:40.443636 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:40.443584 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9njpz" podStartSLOduration=2.5928528870000003 podStartE2EDuration="8.443566417s" totalImagesPullingTime="5.85071353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:40.443403009 +0000 UTC m=+9.615775659" watchObservedRunningTime="2026-11-02 21:33:40.443566417 +0000 UTC m=+9.615939068" Nov 02 21:33:41.045435 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.045398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:41.045435 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.045443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:41.045652 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.045491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:41.045652 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045613 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:41.045652 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045629 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:41.045652 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045638 2582 projected.go:196] Error preparing data for projected volume kube-api-access-bffx9 for pod openshift-network-diagnostics/network-check-target-9xtbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:41.045828 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045742 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9 podName:e987d331-5974-430e-8a6b-d1b1dcbb7fe9 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:49.045708888 +0000 UTC m=+18.218081515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bffx9" (UniqueName: "kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9") pod "network-check-target-9xtbf" (UID: "e987d331-5974-430e-8a6b-d1b1dcbb7fe9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:41.045874 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045830 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:41.045874 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045854 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret podName:4016e9e3-e3e8-454a-a635-49970015c350 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:49.045847344 +0000 UTC m=+18.218219971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret") pod "global-pull-secret-syncer-sr2gb" (UID: "4016e9e3-e3e8-454a-a635-49970015c350") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:41.045942 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045892 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:41.045942 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.045910 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:33:49.045904327 +0000 UTC m=+18.218276953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:41.233613 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.233588 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:41.233772 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.233690 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:41.233772 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.233735 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:41.233907 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.233800 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:41.233907 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.233840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:41.234006 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:41.233917 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:41.338698 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:41.338611 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:42.181023 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:42.180999 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:33:42.245071 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:42.244956 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:33:42.18101893Z","UUID":"f437e7c6-f9ee-48c8-9a54-ce13a9fd50ef","Handler":null,"Name":"","Endpoint":""} Nov 02 21:33:42.246547 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:42.246523 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:33:42.246547 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:42.246552 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:33:42.343340 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:42.343310 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" event={"ID":"fe41410a-c20e-48c0-b3c3-f531c79c3fe5","Type":"ContainerStarted","Data":"93a9f18f37360eaba70653b819754d4fdf8692256b65837e0e00e9ed97f5e03c"} Nov 02 21:33:42.346376 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:42.345933 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbj8m" event={"ID":"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c","Type":"ContainerStarted","Data":"9e8929250e8fa669b8d2cc3a52ec04cd97f0abd9a3ac8c0128c36bbc02ae4a3d"} Nov 02 21:33:43.233533 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.233504 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:43.233689 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:43.233622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:43.233978 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.233955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:43.234072 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:43.234036 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:43.234112 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.234093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:43.234186 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:43.234167 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:43.350863 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.350824 2582 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="52fa4050783a17c6c11f8c7418c350450a913e1c75b8bfa11c8d571e0badcf9a" exitCode=0 Nov 02 21:33:43.351322 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.350891 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerDied","Data":"52fa4050783a17c6c11f8c7418c350450a913e1c75b8bfa11c8d571e0badcf9a"} Nov 02 21:33:43.355036 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.354974 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4qplz" event={"ID":"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a","Type":"ContainerStarted","Data":"1c1f66a757455350cfb83633528ce901269c148b2286fb866c42b828df337c0b"} Nov 02 21:33:43.466818 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:43.465948 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cbj8m" podStartSLOduration=3.178604811 podStartE2EDuration="11.465934708s" totalImagesPullingTime="8.287329897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:42.462437277 +0000 UTC m=+11.634809930" watchObservedRunningTime="2026-11-02 21:33:43.465934708 +0000 UTC m=+12.638307354" Nov 02 21:33:44.358895 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:44.358859 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2nwsl" event={"ID":"d020426a-2902-454c-8b86-7ecc7c666826","Type":"ContainerStarted","Data":"a09f6dfb5eb0e76c6190d826872aa19a7299712d523655dd6f66c861535b240a"} Nov 02 21:33:44.469539 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:44.469493 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4qplz" podStartSLOduration=3.730865167 podStartE2EDuration="12.469479352s" totalImagesPullingTime="8.738614185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:43.576208679 +0000 UTC m=+12.748581328" watchObservedRunningTime="2026-11-02 21:33:44.469479352 +0000 UTC m=+13.641852011" Nov 02 21:33:44.469852 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:44.469820 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2nwsl" podStartSLOduration=3.764548796 podStartE2EDuration="12.469812811s" totalImagesPullingTime="8.705264015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:44.469618087 +0000 UTC m=+13.641990730" watchObservedRunningTime="2026-11-02 21:33:44.469812811 +0000 UTC m=+13.642185498" Nov 02 21:33:45.234010 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.233757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:45.234171 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.233757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:45.234171 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:45.234070 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:45.234171 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:45.234137 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:45.234171 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.233817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:45.234375 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:45.234227 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:45.363190 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.363138 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" event={"ID":"fe41410a-c20e-48c0-b3c3-f531c79c3fe5","Type":"ContainerStarted","Data":"40415d189a6c2da186aab369becf0671e540cd449cc341066169da4a805dbcff"} Nov 02 21:33:45.365021 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.364986 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" event={"ID":"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e","Type":"ContainerStarted","Data":"b6a0f50a1765f729f8cb59a2c95a405692aaea71258b3c8c2c1ef8c3767ed4a6"} Nov 02 21:33:45.367750 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.367725 2582 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="c41b6094c1946d698129bd356c9844421aeb845cbd5131fc7e119171175dbe8e" exitCode=0 Nov 02 21:33:45.368167 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.368139 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerDied","Data":"c41b6094c1946d698129bd356c9844421aeb845cbd5131fc7e119171175dbe8e"} Nov 02 21:33:45.476275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.476229 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" podStartSLOduration=2.974939134 podStartE2EDuration="13.476216426s" totalImagesPullingTime="10.501277292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:45.475699071 +0000 UTC m=+14.648071718" watchObservedRunningTime="2026-11-02 21:33:45.476216426 +0000 UTC m=+14.648589068" Nov 02 21:33:45.706620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:45.706513 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" podStartSLOduration=2.723043267 podStartE2EDuration="13.706495031s" totalImagesPullingTime="10.983451764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:45.706174273 +0000 UTC m=+14.878546923" watchObservedRunningTime="2026-11-02 21:33:45.706495031 +0000 UTC m=+14.878867681" Nov 02 21:33:47.234306 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:47.234219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:47.234306 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:47.234249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:47.234306 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:47.234293 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:47.235092 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:47.234398 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:47.235092 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:47.234511 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:47.235092 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:47.234564 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:47.375219 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:47.375187 2582 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="9eab52650e1633f8312be61e0b48071862a5deccb03773b0e126b323a8cc221b" exitCode=0 Nov 02 21:33:47.375382 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:47.375232 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerDied","Data":"9eab52650e1633f8312be61e0b48071862a5deccb03773b0e126b323a8cc221b"} Nov 02 21:33:49.109379 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.109118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.109411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109297 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.109445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109474 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109490 2582 projected.go:196] Error preparing data for projected volume kube-api-access-bffx9 for pod openshift-network-diagnostics/network-check-target-9xtbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109537 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109552 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9 podName:e987d331-5974-430e-8a6b-d1b1dcbb7fe9 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.109533436 +0000 UTC m=+34.281906081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bffx9" (UniqueName: "kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9") pod "network-check-target-9xtbf" (UID: "e987d331-5974-430e-8a6b-d1b1dcbb7fe9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109563 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109576 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret podName:4016e9e3-e3e8-454a-a635-49970015c350 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.109565145 +0000 UTC m=+34.281937776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret") pod "global-pull-secret-syncer-sr2gb" (UID: "4016e9e3-e3e8-454a-a635-49970015c350") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:49.109951 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.109620 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.109605091 +0000 UTC m=+34.281977722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:49.233513 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.233431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:49.233513 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.233464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:49.233513 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.233463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:49.233771 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.233560 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:49.233771 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.233623 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:49.233771 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:49.233752 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:49.383264 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.383227 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerStarted","Data":"aaedde40f329f6cf0dffcdad69a868c42baa9c4b1b835400b525f689626d776e"} Nov 02 21:33:49.386442 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.386410 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"7e8dc140ce11c61121a23d9a18fb2fdad76376904abefef1f2e30f985dd053c4"} Nov 02 21:33:49.386563 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.386479 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"b84761abfca7044508e272a2d75e0dac3b73fb5588ada02fa30fe9463cdb05ac"} Nov 02 21:33:49.386563 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.386527 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"993162e757adc0595be884418890728b60c003bf64ba7dbb90fdb1c6f1ebe478"} Nov 02 21:33:49.386697 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.386571 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"2164c7d886b8582ffc2157b23e20f9dee5bf105b309d98d97eb20ea0e0216564"} Nov 02 21:33:49.386697 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.386617 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"15aab7425b071f8897aa59dcadb8e252f26af7f6de8d57942b74df0cf7907d82"} Nov 02 21:33:49.386697 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:49.386663 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"1afb5a0f614a8c2549f40eb353afb66fa1f1ead4eb8f4c6644aa582140dad4cd"} Nov 02 21:33:50.390974 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:50.390944 2582 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="aaedde40f329f6cf0dffcdad69a868c42baa9c4b1b835400b525f689626d776e" exitCode=0 Nov 02 21:33:50.391345 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:50.390989 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerDied","Data":"aaedde40f329f6cf0dffcdad69a868c42baa9c4b1b835400b525f689626d776e"} Nov 02 21:33:51.234283 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:51.234244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:51.234448 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:51.234369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:51.234448 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:51.234372 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:51.234550 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:51.234511 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:51.234550 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:51.234546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:51.234654 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:51.234611 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:51.395429 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:51.395348 2582 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="cb9d9a07343438e1a335212d786286bc164ea9b4e4dc10119aa3fffe4a623d39" exitCode=0 Nov 02 21:33:51.395429 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:51.395420 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerDied","Data":"cb9d9a07343438e1a335212d786286bc164ea9b4e4dc10119aa3fffe4a623d39"} Nov 02 21:33:51.398620 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:51.398591 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"6429fe56491948d4223ca6386d2d2f131ffb1fd208bcca9a324861f9cc05a3d8"} Nov 02 21:33:52.404270 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:52.404239 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w9h49" event={"ID":"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3","Type":"ContainerStarted","Data":"7050c4e2242ccfa508524ccd23a322086cce61fdc22bef2286ff129767492de3"} Nov 02 21:33:52.619425 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:52.619382 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w9h49" podStartSLOduration=7.6907851659999995 podStartE2EDuration="20.619368128s" totalImagesPullingTime="12.928582962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:52.570146979 +0000 UTC m=+21.742519641" watchObservedRunningTime="2026-11-02 21:33:52.619368128 +0000 UTC m=+21.791740774" Nov 02 21:33:53.233894 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:53.233859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:53.234035 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:53.233937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:53.234092 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:53.234050 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:53.234092 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:53.234056 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:53.234167 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:53.234132 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:53.234215 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:53.234197 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:53.409174 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:53.409090 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" event={"ID":"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e","Type":"ContainerStarted","Data":"1a41c1ea4c69e0314c2189964c9ce3fa03a5d1248cccd358ac448e7319954fab"} Nov 02 21:33:53.633890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:53.633715 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" podStartSLOduration=6.582433728 podStartE2EDuration="21.633700523s" totalImagesPullingTime="15.051266795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:53.583979709 +0000 UTC m=+22.756352358" watchObservedRunningTime="2026-11-02 21:33:53.633700523 +0000 UTC m=+22.806073172" Nov 02 21:33:54.411616 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:54.411580 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:54.412790 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:54.412765 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:54.412956 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:54.412939 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:54.437857 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:54.437830 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:54.438161 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:54.438143 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:55.233925 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.233890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:55.233925 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.233915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:55.234219 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.233986 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:55.234219 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:55.234063 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:55.234219 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:55.234128 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:55.234219 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:55.234188 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:55.414094 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.414066 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:55.811209 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.811178 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sr2gb"] Nov 02 21:33:55.811361 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.811291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:55.811423 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:55.811401 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:55.814696 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.814661 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9xtbf"] Nov 02 21:33:55.814817 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.814765 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:55.814874 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:55.814858 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:55.815408 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.815389 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tfzjr"] Nov 02 21:33:55.815488 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:55.815476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:55.815574 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:55.815559 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:56.416147 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:56.416116 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:57.234242 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:57.234213 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:57.234521 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:57.234490 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:57.235063 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:57.234262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:57.235063 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:57.234985 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:33:58.233776 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:58.233705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:33:58.234172 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:58.233804 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:33:59.187961 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.187925 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:59.188152 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.188130 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:59.188470 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.188454 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:33:59.212989 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.212969 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:59.213152 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.213141 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:59.223511 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.223491 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:33:59.233795 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.233776 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:33:59.234077 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:33:59.233808 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:33:59.234077 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:59.233902 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xtbf" podUID="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" Nov 02 21:33:59.234077 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:33:59.234026 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfzjr" podUID="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" Nov 02 21:34:00.234235 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:00.234207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:00.234602 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:00.234306 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sr2gb" podUID="4016e9e3-e3e8-454a-a635-49970015c350" Nov 02 21:34:01.094194 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.094007 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeReady" Nov 02 21:34:01.094376 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.094324 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:34:01.206582 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.206554 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lzqpt"] Nov 02 21:34:01.209880 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.209859 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nkwv9"] Nov 02 21:34:01.210002 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.209974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:01.211389 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.211368 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-554b69489-rlbg8"] Nov 02 21:34:01.211507 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.211465 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:01.212720 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.212700 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.233433 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.233416 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:01.233617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.233600 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:01.256644 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.256621 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xxjw8"] Nov 02 21:34:01.258074 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.258058 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9"] Nov 02 21:34:01.258177 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.258163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:01.259873 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.259853 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-kklpk"] Nov 02 21:34:01.260019 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.259974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.261412 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.261395 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw"] Nov 02 21:34:01.261536 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.261522 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.262867 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.262852 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47"] Nov 02 21:34:01.263017 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.262996 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.264289 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.264228 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657747857b-smdws"] Nov 02 21:34:01.264355 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.264342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.265830 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.265814 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv"] Nov 02 21:34:01.265921 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.265910 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.267316 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.267300 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ptr6r"] Nov 02 21:34:01.267423 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.267409 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.269037 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.269020 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h"] Nov 02 21:34:01.269152 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.269131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:01.270947 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.270929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:01.302961 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.302858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrf85\" (UniqueName: \"kubernetes.io/projected/31e0fa56-e920-4006-91c5-7425cd741ead-kube-api-access-xrf85\") pod \"kas-connection-checker-685895c7c9-lzqpt\" (UID: \"31e0fa56-e920-4006-91c5-7425cd741ead\") " pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:01.302961 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.302892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.302961 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.302922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.303131 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303091 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.303131 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.303256 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.303308 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.303371 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.303478 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcdz\" (UniqueName: \"kubernetes.io/projected/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b-kube-api-access-mkcdz\") pod \"kas-connection-checker-685895c7c9-nkwv9\" (UID: \"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b\") " pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:01.303527 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.303499 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.307071 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.307051 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 21:34:01.307368 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.307345 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.310191 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.308321 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.310191 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.308326 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jfs7n\"" Nov 02 21:34:01.310191 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.309877 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-cd7d6d4d5-p5mwz"] Nov 02 21:34:01.314353 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.314332 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nj4zc"] Nov 02 21:34:01.314436 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.314425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.317339 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.317307 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq"] Nov 02 21:34:01.317474 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.317457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.319132 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.319112 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vn9dz"] Nov 02 21:34:01.320065 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.320050 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:01.323157 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.322916 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hsbjx"] Nov 02 21:34:01.323157 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.323118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.325924 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.325902 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lzqpt"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.325938 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nkwv9"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.325961 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554b69489-rlbg8"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.325980 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326000 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xxjw8"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326018 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326032 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h"] Nov 02 21:34:01.326033 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326047 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv"] Nov 02 21:34:01.326580 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326070 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657747857b-smdws"] Nov 02 21:34:01.326580 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326088 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cd7d6d4d5-p5mwz"] Nov 02 21:34:01.326580 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.326129 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.369317 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369293 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 21:34:01.369317 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369313 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 21:34:01.369493 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369326 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 21:34:01.369493 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369340 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.369493 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369293 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 21:34:01.369493 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369303 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s5wdb\"" Nov 02 21:34:01.369493 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369303 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369507 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369517 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369545 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369553 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369660 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369713 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-92ldj\"" Nov 02 21:34:01.369753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369755 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369824 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369826 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369898 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369903 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pwlbm\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.369960 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370017 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x9kkw\"" Nov 02 21:34:01.370060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370061 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370079 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mb8sd\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370089 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370116 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370142 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370228 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370330 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jhcsj\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370337 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370346 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zd85x\"" Nov 02 21:34:01.370357 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370341 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 21:34:01.370778 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370361 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 21:34:01.370778 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370377 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rmsz2\"" Nov 02 21:34:01.370778 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370380 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.370778 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370396 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.370778 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.370462 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.371218 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.371203 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2lcwh\"" Nov 02 21:34:01.404748 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.404724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.404871 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.404759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7kk\" (UniqueName: \"kubernetes.io/projected/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-kube-api-access-zk7kk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.404871 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.404786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.404871 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.404815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.405039 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.404923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrf85\" (UniqueName: \"kubernetes.io/projected/31e0fa56-e920-4006-91c5-7425cd741ead-kube-api-access-xrf85\") pod \"kas-connection-checker-685895c7c9-lzqpt\" (UID: \"31e0fa56-e920-4006-91c5-7425cd741ead\") " pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:01.405039 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.404992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdh9\" (UniqueName: \"kubernetes.io/projected/5af35ed2-0ae8-4679-9c23-419d9a64a2e2-kube-api-access-zcdh9\") pod \"volume-data-source-validator-58c9f5f46d-ctm2h\" (UID: \"5af35ed2-0ae8-4679-9c23-419d9a64a2e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:01.405039 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-trusted-ca\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.405170 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6dbf33-7424-4695-9e9a-6a99475477dd-serving-cert\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.405170 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.405170 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.405170 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405090 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.405170 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-config\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzpg\" (UniqueName: \"kubernetes.io/projected/fc6dbf33-7424-4695-9e9a-6a99475477dd-kube-api-access-5jzpg\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtm7\" (UniqueName: \"kubernetes.io/projected/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-kube-api-access-rmtm7\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f0291ae-a1d6-4312-9e07-da45a493466e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.405360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-snapshots\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.405619 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-stats-auth\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.405619 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.405619 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-service-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.405619 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9vg\" (UniqueName: \"kubernetes.io/projected/ff3812dd-9c0e-482d-b760-4540712ff854-kube-api-access-nn9vg\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.405753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.405753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp982\" (UniqueName: \"kubernetes.io/projected/0f0291ae-a1d6-4312-9e07-da45a493466e-kube-api-access-dp982\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.405753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405688 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-default-certificate\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.405753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.405753 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc6dbf33-7424-4695-9e9a-6a99475477dd-config\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.405890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.405890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4xg\" (UniqueName: \"kubernetes.io/projected/367a2050-9d6e-4486-af6c-1f19921cb8cf-kube-api-access-qk4xg\") pod \"network-check-source-6f78d59f99-bt2cq\" (UID: \"367a2050-9d6e-4486-af6c-1f19921cb8cf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:01.405890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-tmp-dir\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.405890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405850 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b492161-d0aa-42e5-8947-9403654941e3-serving-cert\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.405890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxvm\" (UniqueName: \"kubernetes.io/projected/7b492161-d0aa-42e5-8947-9403654941e3-kube-api-access-rzxvm\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.405890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.406056 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:01.406056 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.406056 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.406056 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.405986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.406056 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.406252 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406073 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.406252 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.406252 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1c67a3a-2cab-4583-976c-a23c07673db0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:01.406252 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406220 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576ks\" (UniqueName: \"kubernetes.io/projected/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-kube-api-access-576ks\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff3812dd-9c0e-482d-b760-4540712ff854-cert\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.406359 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.406373 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vvn\" (UniqueName: \"kubernetes.io/projected/481f04d7-a2a2-472c-8a0b-744c1ca0e486-kube-api-access-z8vvn\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.406415 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.406411 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:01.90639825 +0000 UTC m=+31.078770877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:01.406655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.406655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh29q\" (UniqueName: \"kubernetes.io/projected/06076e5c-0467-47fa-a47b-946e170df5bb-kube-api-access-xh29q\") pod \"kas-connection-checker-685895c7c9-xxjw8\" (UID: \"06076e5c-0467-47fa-a47b-946e170df5bb\") " pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:01.406655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-tmp\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.406655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.406655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.406655 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406601 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82eb2931-ff21-4bf4-9589-995055842cb4-serving-cert\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.406860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.406860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcdz\" (UniqueName: \"kubernetes.io/projected/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b-kube-api-access-mkcdz\") pod \"kas-connection-checker-685895c7c9-nkwv9\" (UID: \"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b\") " pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:01.406860 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mv7c\" (UniqueName: \"kubernetes.io/projected/82eb2931-ff21-4bf4-9589-995055842cb4-kube-api-access-5mv7c\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.406955 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.406869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-config-volume\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.407084 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.407055 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.409910 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.409891 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.423704 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.420190 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 21:34:01.423704 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.422167 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 21:34:01.423704 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.422447 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 21:34:01.423704 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.423576 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw"] Nov 02 21:34:01.425166 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.424244 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 21:34:01.425639 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.425622 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-kklpk"] Nov 02 21:34:01.426843 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.426823 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vn9dz"] Nov 02 21:34:01.426942 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.426888 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.427440 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.427420 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ptr6r"] Nov 02 21:34:01.428097 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.428079 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq"] Nov 02 21:34:01.428705 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.428683 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hsbjx"] Nov 02 21:34:01.428854 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.428839 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.429275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.429260 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nj4zc"] Nov 02 21:34:01.461023 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.461004 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrf85\" (UniqueName: \"kubernetes.io/projected/31e0fa56-e920-4006-91c5-7425cd741ead-kube-api-access-xrf85\") pod \"kas-connection-checker-685895c7c9-lzqpt\" (UID: \"31e0fa56-e920-4006-91c5-7425cd741ead\") " pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:01.461258 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.461241 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.462501 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.462483 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.463012 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.462997 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcdz\" (UniqueName: \"kubernetes.io/projected/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b-kube-api-access-mkcdz\") pod \"kas-connection-checker-685895c7c9-nkwv9\" (UID: \"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b\") " pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:01.463558 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463538 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 21:34:01.463696 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463663 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.463774 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463543 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.463774 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463706 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.463774 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463565 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:34:01.463774 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463725 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.463774 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463539 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:34:01.463774 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463546 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:34:01.464060 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.463776 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 21:34:01.469265 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469246 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tjx6v\"" Nov 02 21:34:01.469360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469339 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plzkt\"" Nov 02 21:34:01.469360 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469262 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:34:01.469472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469279 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 21:34:01.469472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469381 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 21:34:01.469472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469285 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 21:34:01.469472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469260 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:34:01.469472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469281 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 21:34:01.469472 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469401 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5wddx\"" Nov 02 21:34:01.469731 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469615 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7tsg5\"" Nov 02 21:34:01.469731 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469704 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rwnh\"" Nov 02 21:34:01.469993 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.469978 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 21:34:01.508188 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzpg\" (UniqueName: \"kubernetes.io/projected/fc6dbf33-7424-4695-9e9a-6a99475477dd-kube-api-access-5jzpg\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.508275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtm7\" (UniqueName: \"kubernetes.io/projected/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-kube-api-access-rmtm7\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.508275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f0291ae-a1d6-4312-9e07-da45a493466e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.508275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-snapshots\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.508275 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-stats-auth\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.508461 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.508461 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-service-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.508461 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9vg\" (UniqueName: \"kubernetes.io/projected/ff3812dd-9c0e-482d-b760-4540712ff854-kube-api-access-nn9vg\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp982\" (UniqueName: \"kubernetes.io/projected/0f0291ae-a1d6-4312-9e07-da45a493466e-kube-api-access-dp982\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-default-certificate\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc6dbf33-7424-4695-9e9a-6a99475477dd-config\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.508610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4xg\" (UniqueName: \"kubernetes.io/projected/367a2050-9d6e-4486-af6c-1f19921cb8cf-kube-api-access-qk4xg\") pod \"network-check-source-6f78d59f99-bt2cq\" (UID: \"367a2050-9d6e-4486-af6c-1f19921cb8cf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:01.508976 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.508934 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:34:01.509029 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.508984 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls podName:481f04d7-a2a2-472c-8a0b-744c1ca0e486 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.008971851 +0000 UTC m=+31.181344478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-29cmv" (UID: "481f04d7-a2a2-472c-8a0b-744c1ca0e486") : secret "samples-operator-tls" not found Nov 02 21:34:01.509029 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.508984 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-snapshots\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.509136 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509051 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f0291ae-a1d6-4312-9e07-da45a493466e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.509136 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509081 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:01.509230 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509140 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.009123578 +0000 UTC m=+31.181496213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:01.509230 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509146 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:01.509326 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509246 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.009229749 +0000 UTC m=+31.181602375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:01.509326 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-tmp-dir\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.509326 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509285 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-service-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.509326 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509287 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc6dbf33-7424-4695-9e9a-6a99475477dd-config\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.509326 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b492161-d0aa-42e5-8947-9403654941e3-serving-cert\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxvm\" (UniqueName: \"kubernetes.io/projected/7b492161-d0aa-42e5-8947-9403654941e3-kube-api-access-rzxvm\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509504 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509520 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1c67a3a-2cab-4583-976c-a23c07673db0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:01.509564 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.509558 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.009545289 +0000 UTC m=+31.181917940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509563 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-tmp-dir\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576ks\" (UniqueName: \"kubernetes.io/projected/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-kube-api-access-576ks\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff3812dd-9c0e-482d-b760-4540712ff854-cert\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vvn\" (UniqueName: \"kubernetes.io/projected/481f04d7-a2a2-472c-8a0b-744c1ca0e486-kube-api-access-z8vvn\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh29q\" (UniqueName: \"kubernetes.io/projected/06076e5c-0467-47fa-a47b-946e170df5bb-kube-api-access-xh29q\") pod \"kas-connection-checker-685895c7c9-xxjw8\" (UID: \"06076e5c-0467-47fa-a47b-946e170df5bb\") " pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-tmp\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82eb2931-ff21-4bf4-9589-995055842cb4-serving-cert\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mv7c\" (UniqueName: \"kubernetes.io/projected/82eb2931-ff21-4bf4-9589-995055842cb4-kube-api-access-5mv7c\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-config-volume\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.509992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7kk\" (UniqueName: \"kubernetes.io/projected/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-kube-api-access-zk7kk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.510073 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdh9\" (UniqueName: \"kubernetes.io/projected/5af35ed2-0ae8-4679-9c23-419d9a64a2e2-kube-api-access-zcdh9\") pod \"volume-data-source-validator-58c9f5f46d-ctm2h\" (UID: \"5af35ed2-0ae8-4679-9c23-419d9a64a2e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-trusted-ca\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6dbf33-7424-4695-9e9a-6a99475477dd-serving-cert\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510224 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1c67a3a-2cab-4583-976c-a23c07673db0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.510836 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.510308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-config\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.511283 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511051 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-config\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.511283 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511066 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.511386 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.511324 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.011307785 +0000 UTC m=+31.183680432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : configmap references non-existent config key: service-ca.crt Nov 02 21:34:01.511496 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511459 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.511567 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511518 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-stats-auth\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.511567 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.511524 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:01.511567 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511562 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.511753 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.511581 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.011568394 +0000 UTC m=+31.183941038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:01.511815 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511750 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-tmp\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.511815 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.511789 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b492161-d0aa-42e5-8947-9403654941e3-serving-cert\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.511882 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.511921 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.011907398 +0000 UTC m=+31.184280036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512229 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512297 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512335 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-default-certificate\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512567 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-trusted-ca\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512572 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.512617 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512588 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.513108 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.512857 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-config-volume\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.514167 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.514139 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.514362 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.514341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff3812dd-9c0e-482d-b760-4540712ff854-cert\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.514453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.514435 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82eb2931-ff21-4bf4-9589-995055842cb4-serving-cert\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.514739 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.514723 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.514996 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.514978 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6dbf33-7424-4695-9e9a-6a99475477dd-serving-cert\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.520960 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.520944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:01.529628 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.529610 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:01.568436 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.568404 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzpg\" (UniqueName: \"kubernetes.io/projected/fc6dbf33-7424-4695-9e9a-6a99475477dd-kube-api-access-5jzpg\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.570705 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.570613 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.571273 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.571243 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp982\" (UniqueName: \"kubernetes.io/projected/0f0291ae-a1d6-4312-9e07-da45a493466e-kube-api-access-dp982\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:01.572366 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.572043 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7kk\" (UniqueName: \"kubernetes.io/projected/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-kube-api-access-zk7kk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.572366 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.572044 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mv7c\" (UniqueName: \"kubernetes.io/projected/82eb2931-ff21-4bf4-9589-995055842cb4-kube-api-access-5mv7c\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.572366 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.572311 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:01.573038 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.572996 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh29q\" (UniqueName: \"kubernetes.io/projected/06076e5c-0467-47fa-a47b-946e170df5bb-kube-api-access-xh29q\") pod \"kas-connection-checker-685895c7c9-xxjw8\" (UID: \"06076e5c-0467-47fa-a47b-946e170df5bb\") " pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:01.573561 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.573518 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9vg\" (UniqueName: \"kubernetes.io/projected/ff3812dd-9c0e-482d-b760-4540712ff854-kube-api-access-nn9vg\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.573789 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.573708 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtm7\" (UniqueName: \"kubernetes.io/projected/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-kube-api-access-rmtm7\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:01.574063 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.574038 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxvm\" (UniqueName: \"kubernetes.io/projected/7b492161-d0aa-42e5-8947-9403654941e3-kube-api-access-rzxvm\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.574379 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.574362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:01.574660 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.574635 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vvn\" (UniqueName: \"kubernetes.io/projected/481f04d7-a2a2-472c-8a0b-744c1ca0e486-kube-api-access-z8vvn\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:01.575129 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.575106 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576ks\" (UniqueName: \"kubernetes.io/projected/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-kube-api-access-576ks\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:01.575206 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.575189 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdh9\" (UniqueName: \"kubernetes.io/projected/5af35ed2-0ae8-4679-9c23-419d9a64a2e2-kube-api-access-zcdh9\") pod \"volume-data-source-validator-58c9f5f46d-ctm2h\" (UID: \"5af35ed2-0ae8-4679-9c23-419d9a64a2e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:01.575778 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.575763 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4xg\" (UniqueName: \"kubernetes.io/projected/367a2050-9d6e-4486-af6c-1f19921cb8cf-kube-api-access-qk4xg\") pod \"network-check-source-6f78d59f99-bt2cq\" (UID: \"367a2050-9d6e-4486-af6c-1f19921cb8cf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:01.582323 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.582302 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:01.595366 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.595078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:01.636997 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.636964 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:01.649933 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.649307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:01.652557 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.652221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:01.669164 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.668751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:01.868967 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.868945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:01.906976 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.906944 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lzqpt"] Nov 02 21:34:01.909822 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:01.909796 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e0fa56_e920_4006_91c5_7425cd741ead.slice/crio-9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6 WatchSource:0}: Error finding container 9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6: Status 404 returned error can't find the container with id 9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6 Nov 02 21:34:01.916866 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.916837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:01.917149 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.917051 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:01.917149 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.917069 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:01.917149 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:01.917124 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:02.917104817 +0000 UTC m=+32.089477446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:01.952578 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.952551 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9"] Nov 02 21:34:01.955840 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:01.955807 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6dbf33_7424_4695_9e9a_6a99475477dd.slice/crio-6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4 WatchSource:0}: Error finding container 6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4: Status 404 returned error can't find the container with id 6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4 Nov 02 21:34:01.965707 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.965662 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-kklpk"] Nov 02 21:34:01.968733 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:01.968711 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82eb2931_ff21_4bf4_9589_995055842cb4.slice/crio-f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6 WatchSource:0}: Error finding container f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6: Status 404 returned error can't find the container with id f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6 Nov 02 21:34:01.983413 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:01.983366 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47"] Nov 02 21:34:02.008060 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.008023 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a05d0e_bf88_42c3_a8d7_f62e1b039a0f.slice/crio-350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e WatchSource:0}: Error finding container 350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e: Status 404 returned error can't find the container with id 350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e Nov 02 21:34:02.017383 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:02.017564 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:02.017564 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017535 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017512465 +0000 UTC m=+32.189885110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : configmap references non-existent config key: service-ca.crt Nov 02 21:34:02.017734 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:02.017734 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:02.017734 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017572 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:02.017734 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017700 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017686935 +0000 UTC m=+32.190059578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:02.017734 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:02.017734 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017730 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017773 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls podName:481f04d7-a2a2-472c-8a0b-744c1ca0e486 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017759922 +0000 UTC m=+32.190132572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-29cmv" (UID: "481f04d7-a2a2-472c-8a0b-744c1ca0e486") : secret "samples-operator-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.017794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017816 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017837 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017872 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017860504 +0000 UTC m=+32.190233133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017637 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017904 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017895609 +0000 UTC m=+32.190268235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017905 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017934 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017927007 +0000 UTC m=+32.190299638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.017959 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:02.018084 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.018004 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:03.017991228 +0000 UTC m=+32.190363870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:02.018834 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.018810 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nkwv9"] Nov 02 21:34:02.021192 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.021172 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7f8a45_6604_4c5b_9b1d_e761aa0ce16b.slice/crio-2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b WatchSource:0}: Error finding container 2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b: Status 404 returned error can't find the container with id 2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b Nov 02 21:34:02.024430 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.024232 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h"] Nov 02 21:34:02.028011 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.027990 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af35ed2_0ae8_4679_9c23_419d9a64a2e2.slice/crio-8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb WatchSource:0}: Error finding container 8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb: Status 404 returned error can't find the container with id 8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb Nov 02 21:34:02.055184 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.055159 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq"] Nov 02 21:34:02.057847 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.057821 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nj4zc"] Nov 02 21:34:02.058264 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.058239 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367a2050_9d6e_4486_af6c_1f19921cb8cf.slice/crio-9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac WatchSource:0}: Error finding container 9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac: Status 404 returned error can't find the container with id 9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac Nov 02 21:34:02.061169 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.061124 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b492161_d0aa_42e5_8947_9403654941e3.slice/crio-7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7 WatchSource:0}: Error finding container 7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7: Status 404 returned error can't find the container with id 7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7 Nov 02 21:34:02.063059 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.063037 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hsbjx"] Nov 02 21:34:02.066027 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.065996 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3812dd_9c0e_482d_b760_4540712ff854.slice/crio-979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133 WatchSource:0}: Error finding container 979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133: Status 404 returned error can't find the container with id 979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133 Nov 02 21:34:02.230714 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.230685 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xxjw8"] Nov 02 21:34:02.233426 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.233404 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:02.233729 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:02.233701 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06076e5c_0467_47fa_a47b_946e170df5bb.slice/crio-3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a WatchSource:0}: Error finding container 3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a: Status 404 returned error can't find the container with id 3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a Nov 02 21:34:02.285196 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.285177 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fbz7r\"" Nov 02 21:34:02.285891 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.285200 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:34:02.430757 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.430692 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hsbjx" event={"ID":"ff3812dd-9c0e-482d-b760-4540712ff854","Type":"ContainerStarted","Data":"979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133"} Nov 02 21:34:02.432176 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.432148 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" event={"ID":"06076e5c-0467-47fa-a47b-946e170df5bb","Type":"ContainerStarted","Data":"75c54d2985931213531701ed5b9c1ca95412bce1d972e79b2146b0327b627efb"} Nov 02 21:34:02.432286 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.432191 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" event={"ID":"06076e5c-0467-47fa-a47b-946e170df5bb","Type":"ContainerStarted","Data":"3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a"} Nov 02 21:34:02.433305 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.433282 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" event={"ID":"7b492161-d0aa-42e5-8947-9403654941e3","Type":"ContainerStarted","Data":"7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7"} Nov 02 21:34:02.434464 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.434442 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" event={"ID":"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f","Type":"ContainerStarted","Data":"350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e"} Nov 02 21:34:02.435534 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.435514 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" event={"ID":"82eb2931-ff21-4bf4-9589-995055842cb4","Type":"ContainerStarted","Data":"f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6"} Nov 02 21:34:02.436575 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.436555 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" event={"ID":"367a2050-9d6e-4486-af6c-1f19921cb8cf","Type":"ContainerStarted","Data":"9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac"} Nov 02 21:34:02.437658 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.437629 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" event={"ID":"fc6dbf33-7424-4695-9e9a-6a99475477dd","Type":"ContainerStarted","Data":"6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4"} Nov 02 21:34:02.439074 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.439057 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" event={"ID":"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b","Type":"ContainerStarted","Data":"00de1b06f8f221fcff6b825a25e7af2d0218bb92e2719aec5525a643cae1504e"} Nov 02 21:34:02.439147 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.439088 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" event={"ID":"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b","Type":"ContainerStarted","Data":"2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b"} Nov 02 21:34:02.440515 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.440496 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" event={"ID":"31e0fa56-e920-4006-91c5-7425cd741ead","Type":"ContainerStarted","Data":"81a0c6e22fad408e410f89e1d5844903c306790cb870446286e394215a62b843"} Nov 02 21:34:02.440597 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.440526 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" event={"ID":"31e0fa56-e920-4006-91c5-7425cd741ead","Type":"ContainerStarted","Data":"9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6"} Nov 02 21:34:02.441657 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.441636 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" event={"ID":"5af35ed2-0ae8-4679-9c23-419d9a64a2e2","Type":"ContainerStarted","Data":"8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb"} Nov 02 21:34:02.925255 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:02.925218 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:02.925654 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.925635 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:02.925654 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.925656 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:02.925824 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:02.925728 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:04.925710507 +0000 UTC m=+34.098083151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026503 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.026660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.026837 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.026895 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.026878669 +0000 UTC m=+34.199251301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.026975 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.026963625 +0000 UTC m=+34.199336266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : configmap references non-existent config key: service-ca.crt Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027039 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027068 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.027057816 +0000 UTC m=+34.199430445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027125 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027156 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.027145815 +0000 UTC m=+34.199518444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:03.027453 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027208 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:03.028308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027236 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.027227492 +0000 UTC m=+34.199600119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:03.028308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027294 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:34:03.028308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027324 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls podName:481f04d7-a2a2-472c-8a0b-744c1ca0e486 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.027314775 +0000 UTC m=+34.199687405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-29cmv" (UID: "481f04d7-a2a2-472c-8a0b-744c1ca0e486") : secret "samples-operator-tls" not found Nov 02 21:34:03.028308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027388 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:03.028308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027399 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:03.028308 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:03.027426 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:05.027416559 +0000 UTC m=+34.199789188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:03.437890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.436651 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" podStartSLOduration=434.436631531 podStartE2EDuration="7m14.436631531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:03.434688052 +0000 UTC m=+32.607060695" watchObservedRunningTime="2026-11-02 21:34:03.436631531 +0000 UTC m=+32.609004183" Nov 02 21:34:03.437890 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:03.436990 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" podStartSLOduration=434.436980362 podStartE2EDuration="7m14.436980362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:03.322145632 +0000 UTC m=+32.494518279" watchObservedRunningTime="2026-11-02 21:34:03.436980362 +0000 UTC m=+32.609353013" Nov 02 21:34:04.457660 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:04.457629 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" event={"ID":"5af35ed2-0ae8-4679-9c23-419d9a64a2e2","Type":"ContainerStarted","Data":"ebdde190cdb4c7501ce66701ae8353fb2036359b4e9a237ec844dc9918f96563"} Nov 02 21:34:04.570584 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:04.570522 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" podStartSLOduration=435.570502572 podStartE2EDuration="7m15.570502572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:03.547708774 +0000 UTC m=+32.720081443" watchObservedRunningTime="2026-11-02 21:34:04.570502572 +0000 UTC m=+33.742875225" Nov 02 21:34:04.947225 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:04.947185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:04.947533 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:04.947512 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:04.947621 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:04.947563 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:04.947696 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:04.947625 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:08.947605268 +0000 UTC m=+38.119977911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:05.048101 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:05.048269 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:05.048269 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:05.048269 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:05.048423 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:05.048423 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:05.048423 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.048331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:05.048562 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.048481 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:05.048562 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.048547 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.048528899 +0000 UTC m=+38.220901548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:05.048995 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.048975 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.04895781 +0000 UTC m=+38.221330452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : configmap references non-existent config key: service-ca.crt Nov 02 21:34:05.049085 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049065 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:05.049135 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049098 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.049087917 +0000 UTC m=+38.221460558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:05.049208 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049155 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:05.049208 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049186 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.049176026 +0000 UTC m=+38.221548667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:05.049307 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049240 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:05.049307 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049269 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.049260214 +0000 UTC m=+38.221632854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:05.049410 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049322 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:34:05.049410 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049352 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls podName:481f04d7-a2a2-472c-8a0b-744c1ca0e486 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.049343339 +0000 UTC m=+38.221715980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-29cmv" (UID: "481f04d7-a2a2-472c-8a0b-744c1ca0e486") : secret "samples-operator-tls" not found Nov 02 21:34:05.049529 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049412 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:05.049529 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049422 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:05.049529 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.049457 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:09.049440783 +0000 UTC m=+38.221813428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:05.149902 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.149860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:05.150061 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.149994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:05.150061 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.150045 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:05.150997 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.150867 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:34:05.150997 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:05.150935 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:34:37.150916448 +0000 UTC m=+66.323289079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : secret "metrics-daemon-secret" not found Nov 02 21:34:05.154883 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.154832 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:05.157888 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.157859 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:05.244858 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.244645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:05.444465 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.444183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:05.637930 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.637870 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" podStartSLOduration=383.671684115 podStartE2EDuration="6m25.637852069s" totalImagesPullingTime="1.966167954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:04.571470694 +0000 UTC m=+33.743843347" watchObservedRunningTime="2026-11-02 21:34:05.637852069 +0000 UTC m=+34.810224718" Nov 02 21:34:05.638478 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.638230 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sr2gb"] Nov 02 21:34:05.642175 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:05.642144 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4016e9e3_e3e8_454a_a635_49970015c350.slice/crio-328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504 WatchSource:0}: Error finding container 328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504: Status 404 returned error can't find the container with id 328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504 Nov 02 21:34:05.858592 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:05.858515 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9xtbf"] Nov 02 21:34:05.865288 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:05.865253 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode987d331_5974_430e_8a6b_d1b1dcbb7fe9.slice/crio-a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5 WatchSource:0}: Error finding container a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5: Status 404 returned error can't find the container with id a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5 Nov 02 21:34:06.466635 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:06.466594 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sr2gb" event={"ID":"4016e9e3-e3e8-454a-a635-49970015c350","Type":"ContainerStarted","Data":"328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504"} Nov 02 21:34:06.469168 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:06.469147 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/0.log" Nov 02 21:34:06.469283 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:06.469183 2582 generic.go:356] "Generic (PLEG): container finished" podID="82eb2931-ff21-4bf4-9589-995055842cb4" containerID="221231f775a00ab3c0f02528b4c4255e762efa6f3cfcaae9eca625e336e7aa69" exitCode=255 Nov 02 21:34:06.469283 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:06.469261 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" event={"ID":"82eb2931-ff21-4bf4-9589-995055842cb4","Type":"ContainerDied","Data":"221231f775a00ab3c0f02528b4c4255e762efa6f3cfcaae9eca625e336e7aa69"} Nov 02 21:34:06.469518 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:06.469498 2582 scope.go:122] "RemoveContainer" containerID="221231f775a00ab3c0f02528b4c4255e762efa6f3cfcaae9eca625e336e7aa69" Nov 02 21:34:06.471813 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:06.471785 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9xtbf" event={"ID":"e987d331-5974-430e-8a6b-d1b1dcbb7fe9","Type":"ContainerStarted","Data":"a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5"} Nov 02 21:34:07.482959 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.482930 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" event={"ID":"7b492161-d0aa-42e5-8947-9403654941e3","Type":"ContainerStarted","Data":"0ab1721b66640b5863d1ee90c7ac4a27f74cc9708f814f27edd94fcf6608536c"} Nov 02 21:34:07.487907 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.487867 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/1.log" Nov 02 21:34:07.489105 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.488329 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/0.log" Nov 02 21:34:07.489105 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.488366 2582 generic.go:356] "Generic (PLEG): container finished" podID="82eb2931-ff21-4bf4-9589-995055842cb4" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" exitCode=255 Nov 02 21:34:07.489105 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.488406 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" event={"ID":"82eb2931-ff21-4bf4-9589-995055842cb4","Type":"ContainerDied","Data":"18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860"} Nov 02 21:34:07.489105 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.488465 2582 scope.go:122] "RemoveContainer" containerID="221231f775a00ab3c0f02528b4c4255e762efa6f3cfcaae9eca625e336e7aa69" Nov 02 21:34:07.489105 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.488836 2582 scope.go:122] "RemoveContainer" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" Nov 02 21:34:07.489105 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:07.489048 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:07.597915 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:07.597855 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" podStartSLOduration=381.819919094 podStartE2EDuration="6m26.59783718s" totalImagesPullingTime="4.777918086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:07.595313357 +0000 UTC m=+36.767686007" watchObservedRunningTime="2026-11-02 21:34:07.59783718 +0000 UTC m=+36.770209830" Nov 02 21:34:08.278559 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.278527 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b5tqm"] Nov 02 21:34:08.311496 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.311473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.339748 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.339722 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b5tqm"] Nov 02 21:34:08.369967 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.369939 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:34:08.370179 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.370085 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-stcz5\"" Nov 02 21:34:08.370179 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.370143 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:34:08.392206 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.391873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.392206 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.391937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27524\" (UniqueName: \"kubernetes.io/projected/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-api-access-27524\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.392206 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.392004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-crio-socket\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.392206 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.392031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.392206 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.392067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-data-volume\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.492835 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.492798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.493274 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:08.492907 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:08.493274 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.492908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27524\" (UniqueName: \"kubernetes.io/projected/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-api-access-27524\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.493274 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:08.492976 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:08.992955039 +0000 UTC m=+38.165327683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:08.493274 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.493049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-crio-socket\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.493274 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.493078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.493274 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.493115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-data-volume\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.493519 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:08.493392 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:08.493519 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:08.493459 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:08.993434304 +0000 UTC m=+38.165806956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:08.493519 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.493394 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-crio-socket\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.493519 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.493505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-data-volume\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.496213 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.496186 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hsbjx" event={"ID":"ff3812dd-9c0e-482d-b760-4540712ff854","Type":"ContainerStarted","Data":"31f5459fa7b0b2cabf4f9d05107f3896607b30610899b45f53ad74f2cdc9d8c8"} Nov 02 21:34:08.498434 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.498400 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" event={"ID":"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f","Type":"ContainerStarted","Data":"6ff3fbec60d82dd7d51a8279e0d8a12af077968904d93ab5593f18a804966577"} Nov 02 21:34:08.500570 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.500549 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/1.log" Nov 02 21:34:08.500881 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.500860 2582 scope.go:122] "RemoveContainer" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" Nov 02 21:34:08.501235 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:08.501028 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:08.503378 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.503354 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" event={"ID":"367a2050-9d6e-4486-af6c-1f19921cb8cf","Type":"ContainerStarted","Data":"3914dbdc22d8aa3d02b249f84bdeca95bd0093d5abbebb4c94ca2c1d4e4c811c"} Nov 02 21:34:08.507244 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.507222 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" event={"ID":"fc6dbf33-7424-4695-9e9a-6a99475477dd","Type":"ContainerStarted","Data":"41de14bd08fc21d26c7cad0531ce14ff089c7be54de8a94e71dcb3c64d9bce07"} Nov 02 21:34:08.528103 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.528037 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-w59fz-jvbfg"] Nov 02 21:34:08.545506 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.545477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.552446 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.552421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27524\" (UniqueName: \"kubernetes.io/projected/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-api-access-27524\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:08.588155 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.588126 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w59fz-jvbfg"] Nov 02 21:34:08.611361 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.611307 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hsbjx" podStartSLOduration=1.9065803080000001 podStartE2EDuration="7.611290305s" totalImagesPullingTime="5.704709997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:08.610725874 +0000 UTC m=+37.783098564" watchObservedRunningTime="2026-11-02 21:34:08.611290305 +0000 UTC m=+37.783662957" Nov 02 21:34:08.695538 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.695498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.695739 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.695555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.695739 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.695634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.695868 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.695749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.797742 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.796953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.797742 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.797051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.797742 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.797157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.797742 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.797196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.797742 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.797588 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.797742 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.797705 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.798308 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.798280 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.858196 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.858166 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:08.875748 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:08.875716 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.001246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.001362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.001463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.001579 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.001590 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.001634 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.001621163 +0000 UTC m=+46.173993825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.002080 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.002131 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:10.002116299 +0000 UTC m=+39.174488929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.002172 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:09.002527 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.002205 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:10.002194853 +0000 UTC m=+39.174567496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:09.071371 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.071268 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" podStartSLOduration=385.604081598 podStartE2EDuration="6m31.071249687s" totalImagesPullingTime="5.467168089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:09.070930568 +0000 UTC m=+38.243303220" watchObservedRunningTime="2026-11-02 21:34:09.071249687 +0000 UTC m=+38.243622336" Nov 02 21:34:09.073371 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.073326 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" podStartSLOduration=384.66874027 podStartE2EDuration="6m30.073313726s" totalImagesPullingTime="5.404573456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:08.950095225 +0000 UTC m=+38.122467874" watchObservedRunningTime="2026-11-02 21:34:09.073313726 +0000 UTC m=+38.245686376" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.102757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.102875 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.102919 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.102906718 +0000 UTC m=+46.275279359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.102979 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103003 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls podName:481f04d7-a2a2-472c-8a0b-744c1ca0e486 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.102995587 +0000 UTC m=+46.275368227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-29cmv" (UID: "481f04d7-a2a2-472c-8a0b-744c1ca0e486") : secret "samples-operator-tls" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103044 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103051 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103072 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.103065918 +0000 UTC m=+46.275438558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103104 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:09.103280 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103122 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.103116445 +0000 UTC m=+46.275489085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:09.103906 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103162 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.103156951 +0000 UTC m=+46.275529592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : configmap references non-existent config key: service-ca.crt Nov 02 21:34:09.103906 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103194 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:09.103906 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.103205943 +0000 UTC m=+46.275578582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:09.103906 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103243 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:09.103906 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:09.103260 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.103255061 +0000 UTC m=+46.275627701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:09.179434 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.179400 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h"] Nov 02 21:34:09.183356 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.183302 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" podStartSLOduration=382.968417105 podStartE2EDuration="6m29.183284243s" totalImagesPullingTime="6.214867138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:09.182260612 +0000 UTC m=+38.354633262" watchObservedRunningTime="2026-11-02 21:34:09.183284243 +0000 UTC m=+38.355656893" Nov 02 21:34:09.198727 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.198699 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l"] Nov 02 21:34:09.198845 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.198819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:09.227715 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.227690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:09.241754 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.241699 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h"] Nov 02 21:34:09.247195 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.247158 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l"] Nov 02 21:34:09.252110 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.252083 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vkfz5\"" Nov 02 21:34:09.295597 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.295576 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w59fz-jvbfg"] Nov 02 21:34:09.297735 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:09.297703 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1404835_f6fb_400f_b107_476157b43c7d.slice/crio-31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134 WatchSource:0}: Error finding container 31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134: Status 404 returned error can't find the container with id 31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134 Nov 02 21:34:09.304966 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.304944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2h7\" (UniqueName: \"kubernetes.io/projected/99921fd6-6911-4ac5-852d-e90337681ce9-kube-api-access-lc2h7\") pod \"migrator-66df5d496b-p4z7l\" (UID: \"99921fd6-6911-4ac5-852d-e90337681ce9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:09.305228 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.305077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfmr\" (UniqueName: \"kubernetes.io/projected/7cf09ffe-d458-41e7-b9f4-70bf537e64b0-kube-api-access-tmfmr\") pod \"migrator-66df5d496b-jzp7h\" (UID: \"7cf09ffe-d458-41e7-b9f4-70bf537e64b0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:09.331934 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.331912 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:09.332015 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.331986 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:34:09.407530 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.407035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2h7\" (UniqueName: \"kubernetes.io/projected/99921fd6-6911-4ac5-852d-e90337681ce9-kube-api-access-lc2h7\") pod \"migrator-66df5d496b-p4z7l\" (UID: \"99921fd6-6911-4ac5-852d-e90337681ce9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:09.407530 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.407121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfmr\" (UniqueName: \"kubernetes.io/projected/7cf09ffe-d458-41e7-b9f4-70bf537e64b0-kube-api-access-tmfmr\") pod \"migrator-66df5d496b-jzp7h\" (UID: \"7cf09ffe-d458-41e7-b9f4-70bf537e64b0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:09.464877 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.464845 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfmr\" (UniqueName: \"kubernetes.io/projected/7cf09ffe-d458-41e7-b9f4-70bf537e64b0-kube-api-access-tmfmr\") pod \"migrator-66df5d496b-jzp7h\" (UID: \"7cf09ffe-d458-41e7-b9f4-70bf537e64b0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:09.465035 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.464889 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2h7\" (UniqueName: \"kubernetes.io/projected/99921fd6-6911-4ac5-852d-e90337681ce9-kube-api-access-lc2h7\") pod \"migrator-66df5d496b-p4z7l\" (UID: \"99921fd6-6911-4ac5-852d-e90337681ce9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:09.513169 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.512548 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" event={"ID":"f1404835-f6fb-400f-b107-476157b43c7d","Type":"ContainerStarted","Data":"39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb"} Nov 02 21:34:09.513169 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.512605 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" event={"ID":"f1404835-f6fb-400f-b107-476157b43c7d","Type":"ContainerStarted","Data":"31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134"} Nov 02 21:34:09.532098 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.531919 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:09.558199 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.558175 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:09.629417 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.629364 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" podStartSLOduration=1.6293440829999999 podStartE2EDuration="1.629344083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:09.627800492 +0000 UTC m=+38.800173143" watchObservedRunningTime="2026-11-02 21:34:09.629344083 +0000 UTC m=+38.801716733" Nov 02 21:34:09.931339 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.931313 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h"] Nov 02 21:34:09.933709 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:09.933666 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf09ffe_d458_41e7_b9f4_70bf537e64b0.slice/crio-ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099 WatchSource:0}: Error finding container ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099: Status 404 returned error can't find the container with id ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099 Nov 02 21:34:09.956610 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.956588 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l"] Nov 02 21:34:09.958498 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:09.958466 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99921fd6_6911_4ac5_852d_e90337681ce9.slice/crio-e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133 WatchSource:0}: Error finding container e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133: Status 404 returned error can't find the container with id e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133 Nov 02 21:34:09.988504 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:09.988473 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tzk7v"] Nov 02 21:34:10.014746 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.014719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:10.014898 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.014800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:10.014898 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:10.014894 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:10.015007 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:10.014914 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:10.015007 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:10.014952 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:12.014932601 +0000 UTC m=+41.187305250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:10.015121 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:10.015047 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:12.01503557 +0000 UTC m=+41.187408215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:10.016580 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.016553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.048906 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.048880 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tzk7v"] Nov 02 21:34:10.070142 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.069931 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 21:34:10.070142 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.069931 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 21:34:10.070142 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.070033 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 21:34:10.070142 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.070086 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 21:34:10.070142 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.070093 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zfnt5\"" Nov 02 21:34:10.115631 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.115584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pztg\" (UniqueName: \"kubernetes.io/projected/0f6b3b2e-5e40-4067-9e29-a891a598286f-kube-api-access-2pztg\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.115807 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.115769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-key\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.115942 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.115915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-cabundle\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.217120 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.217034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-cabundle\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.217271 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.217179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pztg\" (UniqueName: \"kubernetes.io/projected/0f6b3b2e-5e40-4067-9e29-a891a598286f-kube-api-access-2pztg\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.217337 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.217307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-key\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.217896 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.217873 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-cabundle\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.220165 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.220132 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-key\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.274332 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.274306 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pztg\" (UniqueName: \"kubernetes.io/projected/0f6b3b2e-5e40-4067-9e29-a891a598286f-kube-api-access-2pztg\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.342942 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.342905 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:10.518706 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.518592 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9xtbf" event={"ID":"e987d331-5974-430e-8a6b-d1b1dcbb7fe9","Type":"ContainerStarted","Data":"20bb1b4fe8a522cbbb78609a4a744b8f1edf6f75271aff36aeb397201fc6692d"} Nov 02 21:34:10.519158 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.518790 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:10.519937 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.519903 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" event={"ID":"99921fd6-6911-4ac5-852d-e90337681ce9","Type":"ContainerStarted","Data":"e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133"} Nov 02 21:34:10.521187 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.521164 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" event={"ID":"7cf09ffe-d458-41e7-b9f4-70bf537e64b0","Type":"ContainerStarted","Data":"ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099"} Nov 02 21:34:10.632875 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.632831 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9xtbf" podStartSLOduration=34.98750993 podStartE2EDuration="38.632819379s" totalImagesPullingTime="3.645309449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:10.632586734 +0000 UTC m=+39.804959385" watchObservedRunningTime="2026-11-02 21:34:10.632819379 +0000 UTC m=+39.805192028" Nov 02 21:34:10.735551 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:10.735507 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tzk7v"] Nov 02 21:34:10.742744 ip-10-0-128-36 kubenswrapper[2582]: W1102 21:34:10.742716 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6b3b2e_5e40_4067_9e29_a891a598286f.slice/crio-eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d WatchSource:0}: Error finding container eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d: Status 404 returned error can't find the container with id eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d Nov 02 21:34:11.528321 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:11.528063 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" event={"ID":"0f6b3b2e-5e40-4067-9e29-a891a598286f","Type":"ContainerStarted","Data":"061c89bf9169b3413894b88c841a31c292fcfd6a23cfb09fa6c4263c3e1deaaf"} Nov 02 21:34:11.528321 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:11.528116 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" event={"ID":"0f6b3b2e-5e40-4067-9e29-a891a598286f","Type":"ContainerStarted","Data":"eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d"} Nov 02 21:34:11.583124 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:11.583088 2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:11.583304 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:11.583156 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:11.583596 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:11.583578 2582 scope.go:122] "RemoveContainer" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" Nov 02 21:34:11.583868 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:11.583844 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:12.035210 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.035176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:12.035350 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:12.035303 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:12.035386 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:12.035362 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:16.035348649 +0000 UTC m=+45.207721276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:12.035386 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.035309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:12.035465 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:12.035378 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:12.035504 ip-10-0-128-36 kubenswrapper[2582]: E1102 21:34:12.035477 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:16.035452806 +0000 UTC m=+45.207825436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:12.242681 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.242651 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4qplz_557fdf7f-9d7a-4de7-9819-becbc1fbfb8a/dns-node-resolver/0.log" Nov 02 21:34:12.536704 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.536312 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" event={"ID":"99921fd6-6911-4ac5-852d-e90337681ce9","Type":"ContainerStarted","Data":"e4d2d58bb33a7188c98ca3e9fa10d08a0c8bdb46c8e2f85f212beee178e9a842"} Nov 02 21:34:12.542961 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.542884 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" event={"ID":"7cf09ffe-d458-41e7-b9f4-70bf537e64b0","Type":"ContainerStarted","Data":"75d81aee72bdeb8d44bcf793617d9a6ce5c79798dfd327112eefe880bd40dc71"} Nov 02 21:34:12.654544 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.654186 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" podStartSLOduration=1.4485953280000001 podStartE2EDuration="3.654167664s" totalImagesPullingTime="2.205572336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:12.653480885 +0000 UTC m=+41.825853537" watchObservedRunningTime="2026-11-02 21:34:12.654167664 +0000 UTC m=+41.826540314" Nov 02 21:34:12.654876 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.654834 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" podStartSLOduration=3.654822962 podStartE2EDuration="3.654822962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:11.687659976 +0000 UTC m=+40.860032622" watchObservedRunningTime="2026-11-02 21:34:12.654822962 +0000 UTC m=+41.827195611" Nov 02 21:34:12.768281 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.768172 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" podStartSLOduration=1.5402569769999999 podStartE2EDuration="3.768152775s" totalImagesPullingTime="2.227895798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:12.765986568 +0000 UTC m=+41.938359215" watchObservedRunningTime="2026-11-02 21:34:12.768152775 +0000 UTC m=+41.940525425" Nov 02 21:34:12.776827 ip-10-0-128-36 kubenswrapper[2582]: I1102 21:34:12.776756 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:34:12.776945 ip-10-0-128-36 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:34:12.789115 ip-10-0-128-36 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:34:12.789336 ip-10-0-128-36 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:34:12.789574 ip-10-0-128-36 systemd[1]: kubelet.service: Consumed 3.127s CPU time. Nov 02 21:34:12.797296 ip-10-0-128-36 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:34:12.887550 ip-10-0-128-36 kubenswrapper[5091]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:34:12.887550 ip-10-0-128-36 kubenswrapper[5091]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:34:12.887550 ip-10-0-128-36 kubenswrapper[5091]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:34:12.887550 ip-10-0-128-36 kubenswrapper[5091]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:34:12.889840 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889763 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.889840 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889837 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.889840 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889841 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.889840 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889846 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889851 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889855 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889858 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889862 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889866 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889868 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889879 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889883 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889885 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889888 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889890 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889893 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889896 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889899 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889901 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889904 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889906 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889909 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.890007 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889912 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889915 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889918 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889920 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889923 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889929 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889931 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889934 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889937 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889939 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889942 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889945 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889947 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889951 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889953 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889956 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889959 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889961 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889964 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889966 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.890492 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889969 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889973 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889975 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889978 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889981 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889983 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889986 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889989 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889991 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889994 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889996 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.889999 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890001 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890004 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890007 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890010 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890013 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890017 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890020 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.891013 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890022 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890025 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890028 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890030 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890033 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890036 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890038 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890041 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890044 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890046 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890049 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890051 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890054 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890056 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890059 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890062 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890065 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890067 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890070 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890072 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.891749 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890075 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890078 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890080 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890083 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890085 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890089 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890091 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890094 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890097 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890099 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890103 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890108 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890113 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890116 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890119 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890121 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890124 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890127 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890129 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890132 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.892570 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890134 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890137 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890140 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890142 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.890145 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891022 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891030 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891033 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891042 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891045 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891048 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891051 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891054 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891056 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891059 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891062 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891064 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891069 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891071 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.893231 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891074 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891078 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891082 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891087 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891090 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891191 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891385 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891395 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891402 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891409 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891414 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891419 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891424 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891428 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891432 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891437 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891441 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891445 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891456 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.893751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891461 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891465 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891469 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891477 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891485 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891489 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891494 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891499 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891503 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891508 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891513 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891523 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891527 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891532 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891537 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891541 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891546 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891550 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891554 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.894255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891559 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891581 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891586 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891590 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891594 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891604 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891608 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891612 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891616 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891621 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891629 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891633 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891638 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891643 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891647 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891651 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891655 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891666 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891685 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891689 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.894748 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891694 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891699 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891703 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891708 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891712 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891716 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891720 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891725 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891729 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891734 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891742 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891747 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891752 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891756 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891778 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891783 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891787 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891791 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891795 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891802 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.895254 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891806 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891811 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891819 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891824 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891828 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891833 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891837 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891854 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891858 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891862 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891867 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891871 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891876 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.891880 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893832 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893850 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893857 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893863 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893867 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.895783 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893872 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893876 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893881 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893885 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893890 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893894 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893898 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893902 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893906 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893910 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893914 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893918 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893923 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893927 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893931 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893936 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893940 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893944 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893949 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893953 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.896242 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893957 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893961 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893965 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893969 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893973 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893978 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893982 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893986 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893990 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893994 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.893998 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894002 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894006 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894010 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894014 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894018 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894023 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894027 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894031 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.896753 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894035 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894039 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894043 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894047 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894051 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894056 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894060 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894064 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894068 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894072 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894076 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894081 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894085 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894090 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894094 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894099 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894103 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894107 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894110 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894115 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.897216 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894120 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894124 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894128 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894134 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894140 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894144 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894149 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894153 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894157 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894161 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894165 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894170 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894174 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894178 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894182 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894186 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894191 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894195 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894199 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.897733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894203 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894208 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894212 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894216 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894223 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894227 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894231 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894235 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894239 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894243 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894247 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894259 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894264 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894268 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894273 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894277 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894282 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894286 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894290 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894294 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.898199 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894298 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894302 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.894306 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894428 5091 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894437 5091 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894451 5091 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894458 5091 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894464 5091 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894469 5091 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894476 5091 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894486 5091 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894491 5091 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894496 5091 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894501 5091 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894506 5091 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894511 5091 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894516 5091 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894521 5091 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894525 5091 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894530 5091 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894535 5091 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894543 5091 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894548 5091 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:34:12.898797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894552 5091 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894565 5091 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894570 5091 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894576 5091 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894581 5091 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894585 5091 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894590 5091 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894595 5091 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894600 5091 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894605 5091 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894610 5091 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894614 5091 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894620 5091 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894626 5091 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894630 5091 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894635 5091 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894640 5091 flags.go:64] FLAG: --enable-server="true" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894644 5091 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894655 5091 flags.go:64] FLAG: --event-burst="100" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894660 5091 flags.go:64] FLAG: --event-qps="50" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894664 5091 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894691 5091 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894697 5091 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894704 5091 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:34:12.899374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894708 5091 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894714 5091 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894719 5091 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894724 5091 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894728 5091 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894733 5091 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894737 5091 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894741 5091 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894745 5091 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894750 5091 flags.go:64] FLAG: --feature-gates="" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894766 5091 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894771 5091 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894776 5091 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894781 5091 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894785 5091 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894790 5091 flags.go:64] FLAG: --help="false" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894795 5091 flags.go:64] FLAG: --hostname-override="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894800 5091 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894805 5091 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894809 5091 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894815 5091 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894820 5091 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894825 5091 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:34:12.899973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894829 5091 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894834 5091 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894838 5091 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894843 5091 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894848 5091 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894852 5091 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894858 5091 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894862 5091 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894867 5091 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894872 5091 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894876 5091 flags.go:64] FLAG: --lock-file="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894881 5091 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894885 5091 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894890 5091 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894898 5091 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894903 5091 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894908 5091 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894912 5091 flags.go:64] FLAG: --logging-format="text" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894917 5091 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894922 5091 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894936 5091 flags.go:64] FLAG: --manifest-url="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894940 5091 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894947 5091 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894952 5091 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894958 5091 flags.go:64] FLAG: --max-pods="110" Nov 02 21:34:12.900623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894966 5091 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894971 5091 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894976 5091 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894980 5091 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894985 5091 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894990 5091 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.894995 5091 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895005 5091 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895010 5091 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895015 5091 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895020 5091 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895024 5091 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895029 5091 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895034 5091 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895038 5091 flags.go:64] FLAG: --port="10250" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895043 5091 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895047 5091 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0cff16335a8165556" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895052 5091 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895057 5091 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895062 5091 flags.go:64] FLAG: --register-node="true" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895066 5091 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895072 5091 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895076 5091 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895081 5091 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895085 5091 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895091 5091 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:34:12.901259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895096 5091 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895101 5091 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895113 5091 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895117 5091 flags.go:64] FLAG: --runonce="false" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895122 5091 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895127 5091 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895133 5091 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895138 5091 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895143 5091 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895149 5091 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895154 5091 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895159 5091 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895164 5091 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895168 5091 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895173 5091 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895177 5091 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895182 5091 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895187 5091 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895191 5091 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895200 5091 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895204 5091 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895209 5091 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895218 5091 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895222 5091 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895227 5091 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:34:12.901920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895231 5091 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895236 5091 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895240 5091 flags.go:64] FLAG: --v="2" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895247 5091 flags.go:64] FLAG: --version="false" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895253 5091 flags.go:64] FLAG: --vmodule="" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895259 5091 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.895264 5091 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895491 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895498 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895503 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895516 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895522 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895526 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895532 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895537 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895541 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895547 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895551 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895555 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895561 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895567 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.902528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895572 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895577 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895581 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895586 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895590 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895594 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895598 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895602 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895607 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895611 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895615 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895619 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895623 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895627 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895631 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895636 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895640 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895644 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895648 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.903057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895652 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895656 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895660 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895684 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895690 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895696 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895701 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895705 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895711 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895715 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895720 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895724 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895728 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895732 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895736 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895740 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895744 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895749 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895753 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895757 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.903522 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895761 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895766 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895770 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895774 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895778 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895782 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895787 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895791 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895795 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895799 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895803 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895808 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895812 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895816 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895820 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895824 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895835 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895841 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895846 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895850 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.904133 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895855 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895859 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895864 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895868 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895872 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895876 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895880 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895884 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895888 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895892 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895896 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895901 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895905 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895909 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895913 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895917 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895921 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895926 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895930 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895934 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.904633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895939 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895943 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895947 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895951 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895956 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895960 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895964 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895970 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895976 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895985 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895992 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.895997 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896004 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.896042 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896105 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.905149 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896111 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896116 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896120 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896125 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896129 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896133 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896137 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896142 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896146 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896150 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896154 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896158 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896162 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896167 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896171 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896176 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896180 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896184 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896188 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896192 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.905722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896196 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896200 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896205 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896209 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896213 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896218 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896222 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896228 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896232 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896239 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896243 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896247 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896252 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896256 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896260 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896264 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896268 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896272 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896277 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.906504 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896281 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896285 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896289 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896294 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896298 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896303 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896307 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896311 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896315 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896320 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896324 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896328 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896332 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896336 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896340 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896344 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896349 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896353 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896357 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896361 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.907054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896366 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896371 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896376 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896380 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896384 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896388 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896393 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896397 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896401 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896405 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896409 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896414 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896419 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896423 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896429 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896436 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896441 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896446 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896450 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.907886 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896455 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896459 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896463 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896467 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896471 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896475 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896479 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896484 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896488 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896492 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896496 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896500 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896504 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896510 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896516 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896522 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896527 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896531 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896535 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.908751 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896541 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896545 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896549 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896553 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896557 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896562 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896566 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.896570 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.901026 5091 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.901040 5091 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.901060 5091 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.901065 5091 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901117 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901122 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901125 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901129 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901132 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901136 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901141 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.909360 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901145 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901148 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901151 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901154 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901157 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901160 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901162 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901165 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901168 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901170 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901173 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901175 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901178 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901181 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901184 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901186 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901189 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901191 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901194 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901196 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.910039 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901199 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901202 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901206 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901209 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901212 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901214 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901217 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901219 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901222 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901225 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901227 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901230 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901232 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901234 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901237 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901240 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901243 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901245 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901248 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901250 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.910609 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901253 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901255 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901258 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901260 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901263 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901267 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901269 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901272 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901275 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901278 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901280 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901283 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901285 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901288 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901290 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901293 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901296 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901298 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901301 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.911161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901303 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901306 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901308 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901311 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901314 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901317 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901319 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901322 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901324 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901327 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901331 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901334 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901337 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901340 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901343 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901346 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901348 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901351 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901353 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901356 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.911618 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901358 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901361 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901363 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901366 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901368 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901371 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901374 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901376 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901379 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901383 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901386 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901389 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901392 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901394 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901397 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901399 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901402 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901404 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901407 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.912113 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901409 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.901436 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901589 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901595 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901598 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901601 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901605 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901608 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901612 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901615 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901618 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901621 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901623 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901626 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901629 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901631 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.912574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901634 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901636 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901639 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901641 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901644 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901646 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901649 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901652 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901655 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901658 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901661 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901665 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901689 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901692 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901695 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901698 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901700 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901703 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901706 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.913067 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901708 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901711 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901713 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901716 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901718 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901722 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901724 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901727 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901730 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901732 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901735 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901738 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901740 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901743 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901745 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901748 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901751 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901754 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901756 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901759 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.913691 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901761 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901765 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901768 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901771 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901773 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901776 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901778 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901781 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901783 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901786 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901789 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901792 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901794 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901797 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901800 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901802 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901805 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901807 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901810 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901812 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.914324 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901817 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901820 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901823 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901826 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901829 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901831 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901834 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901836 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901839 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901842 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901844 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901847 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901850 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901852 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901855 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901858 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901861 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901863 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901866 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.914916 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901868 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901871 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901874 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901877 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901879 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901882 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901885 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901887 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901890 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901892 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901895 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901897 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901900 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901902 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.901926 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901958 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:12.915407 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901962 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901965 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901968 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901971 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901973 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901976 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901979 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901981 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901987 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901992 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901995 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.901998 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902001 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902004 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902007 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902010 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902013 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902016 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902018 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:12.916436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902021 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902024 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902027 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902029 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902032 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902035 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902037 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902040 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902042 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902045 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902047 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902050 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902052 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902055 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902058 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902060 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902063 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902065 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902068 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902070 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:12.917223 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902073 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902078 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902080 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902083 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902087 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902090 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902093 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902096 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902098 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902101 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902103 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902106 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902109 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902111 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902114 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902116 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902119 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902122 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902126 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:12.918090 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902129 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902132 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902135 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902137 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902139 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902142 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902145 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902147 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902150 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902152 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902155 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902157 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902160 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902162 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902166 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902169 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902171 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902176 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902179 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:12.918955 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902181 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902184 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902187 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902190 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902192 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902195 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902197 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902200 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902202 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902205 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902207 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902210 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902213 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902215 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902217 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902220 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902223 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902225 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902228 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902230 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:12.919741 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902233 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902236 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902239 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902241 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902244 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902246 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902249 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:12.902253 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.902302 5091 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903243 5091 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903293 5091 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903480 5091 server.go:1014] "Starting client certificate rotation" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903624 5091 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903683 5091 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:28:31 +0000 UTC" deadline="2028-05-20 01:33:20.078088737 +0000 UTC" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903729 5091 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13539h59m7.17436279s" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.903936 5091 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:34:12.920551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.904906 5091 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:34:12.921208 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.906747 5091 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:34:12.921208 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.909020 5091 log.go:25] "Validated CRI v1 image API" Nov 02 21:34:12.921208 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.909899 5091 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:34:12.921208 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.911831 5091 fs.go:135] Filesystem UUIDs: map[49e0b128-9751-4971-9fa9-1264b3343913:/dev/nvme0n1p3 58667e7a-9569-4cbd-88b9-b12113450898:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 21:34:12.921401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.911845 5091 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06076e5c-0467-47fa-a47b-946e170df5bb/volumes/kubernetes.io~projected/kube-api-access-xh29q:{mountpoint:/var/lib/kubelet/pods/06076e5c-0467-47fa-a47b-946e170df5bb/volumes/kubernetes.io~projected/kube-api-access-xh29q major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0a36a5-f22c-4759-8e09-724db23b7c04/volumes/kubernetes.io~projected/kube-api-access-8s6pr:{mountpoint:/var/lib/kubelet/pods/0d0a36a5-f22c-4759-8e09-724db23b7c04/volumes/kubernetes.io~projected/kube-api-access-8s6pr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f0291ae-a1d6-4312-9e07-da45a493466e/volumes/kubernetes.io~projected/kube-api-access-dp982:{mountpoint:/var/lib/kubelet/pods/0f0291ae-a1d6-4312-9e07-da45a493466e/volumes/kubernetes.io~projected/kube-api-access-dp982 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f6b3b2e-5e40-4067-9e29-a891a598286f/volumes/kubernetes.io~projected/kube-api-access-2pztg:{mountpoint:/var/lib/kubelet/pods/0f6b3b2e-5e40-4067-9e29-a891a598286f/volumes/kubernetes.io~projected/kube-api-access-2pztg major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f6b3b2e-5e40-4067-9e29-a891a598286f/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0f6b3b2e-5e40-4067-9e29-a891a598286f/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~projected/kube-api-access-rmtm7:{mountpoint:/var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~projected/kube-api-access-rmtm7 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e0fa56-e920-4006-91c5-7425cd741ead/volumes/kubernetes.io~projected/kube-api-access-xrf85:{mountpoint:/var/lib/kubelet/pods/31e0fa56-e920-4006-91c5-7425cd741ead/volumes/kubernetes.io~projected/kube-api-access-xrf85 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367a2050-9d6e-4486-af6c-1f19921cb8cf/volumes/kubernetes.io~projected/kube-api-access-qk4xg:{mountpoint:/var/lib/kubelet/pods/367a2050-9d6e-4486-af6c-1f19921cb8cf/volumes/kubernetes.io~projected/kube-api-access-qk4xg major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4016e9e3-e3e8-454a-a635-49970015c350/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4016e9e3-e3e8-454a-a635-49970015c350/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/481f04d7-a2a2-472c-8a0b-744c1ca0e486/volumes/kubernetes.io~projected/kube-api-access-z8vvn:{mountpoint:/var/lib/kubelet/pods/481f04d7-a2a2-472c-8a0b-744c1ca0e486/volumes/kubernetes.io~projected/kube-api-access-z8vvn major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a/volumes/kubernetes.io~projected/kube-api-access-rnnd2:{mountpoint:/var/lib/kubelet/pods/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a/volumes/kubernetes.io~projected/kube-api-access-rnnd2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3/volumes/kubernetes.io~projected/kube-api-access-fw7rx:{mountpoint:/var/lib/kubelet/pods/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3/volumes/kubernetes.io~projected/kube-api-access-fw7rx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5af35ed2-0ae8-4679-9c23-419d9a64a2e2/volumes/kubernetes.io~projected/kube-api-access-zcdh9:{mountpoint:/var/lib/kubelet/pods/5af35ed2-0ae8-4679-9c23-419d9a64a2e2/volumes/kubernetes.io~projected/kube-api-access-zcdh9 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volumes/kubernetes.io~projected/kube-api-access-c5dgc:{mountpoint:/var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volumes/kubernetes.io~projected/kube-api-access-c5dgc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6/volumes/kubernetes.io~projected/kube-api-access-576ks:{mountpoint:/var/lib/kubelet/pods/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6/volumes/kubernetes.io~projected/kube-api-access-576ks major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b492161-d0aa-42e5-8947-9403654941e3/volumes/kubernetes.io~projected/kube-api-access-rzxvm:{mountpoint:/var/lib/kubelet/pods/7b492161-d0aa-42e5-8947-9403654941e3/volumes/kubernetes.io~projected/kube-api-access-rzxvm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b492161-d0aa-42e5-8947-9403654941e3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7b492161-d0aa-42e5-8947-9403654941e3/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cf09ffe-d458-41e7-b9f4-70bf537e64b0/volumes/kubernetes.io~projected/kube-api-access-tmfmr:{mountpoint:/var/lib/kubelet/pods/7cf09ffe-d458-41e7-b9f4-70bf537e64b0/volumes/kubernetes.io~projected/kube-api-access-tmfmr major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b/volumes/kubernetes.io~projected/kube-api-access-mkcdz:{mountpoint:/var/lib/kubelet/pods/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b/volumes/kubernetes.io~projected/kube-api-access-mkcdz major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82eb2931-ff21-4bf4-9589-995055842cb4/volumes/kubernetes.io~projected/kube-api-access-5mv7c:{mountpoint:/var/lib/kubelet/pods/82eb2931-ff21-4bf4-9589-995055842cb4/volumes/kubernetes.io~projected/kube-api-access-5mv7c major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82eb2931-ff21-4bf4-9589-995055842cb4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/82eb2931-ff21-4bf4-9589-995055842cb4/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~projected/kube-api-access-22hsk:{mountpoint:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~projected/kube-api-access-22hsk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/volumes/kubernetes.io~projected/kube-api-access-zk7kk:{mountpoint:/var/lib/kubelet/pods/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/volumes/kubernetes.io~projected/kube-api-access-zk7kk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99921fd6-6911-4ac5-852d-e90337681ce9/volumes/kubernetes.io~projected/kube-api-access-lc2h7:{mountpoint:/var/lib/kubelet/pods/99921fd6-6911-4ac5-852d-e90337681ce9/volumes/kubernetes.io~projected/kube-api-access-lc2h7 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3900b59-124a-4dc5-9cb4-e5e98253eb5a/volumes/kubernetes.io~projected/kube-api-access-rp8kt:{mountpoint:/var/lib/kubelet/pods/b3900b59-124a-4dc5-9cb4-e5e98253eb5a/volumes/kubernetes.io~projected/kube-api-access-rp8kt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c/volumes/kubernetes.io~projected/kube-api-access-4vz8v:{mountpoint:/var/lib/kubelet/pods/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c/volumes/kubernetes.io~projected/kube-api-access-4vz8v major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a2b13a-685e-4ba3-8370-e059d7fe5e26/volumes/kubernetes.io~projected/kube-api-access-27524:{mountpoint:/var/lib/kubelet/pods/c5a2b13a-685e-4ba3-8370-e059d7fe5e26/volumes/kubernetes.io~projected/kube-api-access-27524 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d020426a-2902-454c-8b86-7ecc7c666826/volumes/kubernetes.io~projected/kube-api-access-qm6dc:{mountpoint:/var/lib/kubelet/pods/d020426a-2902-454c-8b86-7ecc7c666826/volumes/kubernetes.io~projected/kube-api-access-qm6dc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e987d331-5974-430e-8a6b-d1b1dcbb7fe9/volumes/kubernetes.io~projected/kube-api-access-bffx9:{mountpoint:/var/lib/kubelet/pods/e987d331-5974-430e-8a6b-d1b1dcbb7fe9/volumes/kubernetes.io~projected/kube-api-access-bffx9 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb533acd-73cb-4f30-a60e-e96debb69f25/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/eb533acd-73cb-4f30-a60e-e96debb69f25/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1404835-f6fb-400f-b107-476157b43c7d/volumes/kubernetes.io~projected/kube-api-access-fkl8p:{mountpoint:/var/lib/kubelet/pods/f1404835-f6fb-400f-b107-476157b43c7d/volumes/kubernetes.io~projected/kube-api-access-fkl8p major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~projected/kube-api-access-n78t6:{mountpoint:/var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~projected/kube-api-access-n78t6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc6dbf33-7424-4695-9e9a-6a99475477dd/volumes/kubernetes.io~projected/kube-api-access-5jzpg:{mountpoint:/var/lib/kubelet/pods/fc6dbf33-7424-4695-9e9a-6a99475477dd/volumes/kubernetes.io~projected/kube-api-access-5jzpg major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc6dbf33-7424-4695-9e9a-6a99475477dd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fc6dbf33-7424-4695-9e9a-6a99475477dd/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~projected/kube-api-access-ghzsm:{mountpoint:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~projected/kube-api-access-ghzsm major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe41410a-c20e-48c0-b3c3-f531c79c3fe5/volumes/kubernetes.io~projected/kube-api-access-mlqts:{mountpoint:/var/lib/kubelet/pods/fe41410a-c20e-48c0-b3c3-f531c79c3fe5/volumes/kubernetes.io~projected/kube-api-access-mlqts major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff3812dd-9c0e-482d-b760-4540712ff854/volumes/kubernetes.io~projected/kube-api-access-nn9vg:{mountpoint:/var/lib/kubelet/pods/ff3812dd-9c0e-482d-b760-4540712ff854/volumes/kubernetes.io~projected/kube-api-access-nn9vg major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff3812dd-9c0e-482d-b760-4540712ff854/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ff3812dd-9c0e-482d-b760-4540712ff854/volumes/kubernetes.io~secret/cert major:0 minor:209 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d36dd7a77ddcdc8506c552c901a5b3018ab01b6470b969551edf5fbecd136ea9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ef72fcde516ed729568a68daac6312b014040b7abb03caac450934ff8540f207/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a847e2c0bb68d3023c59fe4a9e27d82c3cc2fd81875e326eda0b96ea9f9e6da8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7f4e15aa25e6059a84b361c73dfa668213a6575fab5bfcab437a05d3e833fb5e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/48af68832c4ffd3fc4807d9ff6deb11612b7cf752d945c4e142354e7a56a99ec/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e747eed2ac16b0d86a581a685adcb06054dbd062acf85012644e9a887570201f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3680e4d36c03a3bb3e448e308e71ae182b1008aa6dca9d24c99969d877a2b992/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/213f2ad00918717119fdbfc7005d8f90e56384be46d13c9c1cf3de83c01245f2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a8ed756c01ad66cedbd93d940934c873a53fe6c44666df1a1b6cdd08f855757b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c551fee54795f5701c7802da3611dcc4b7d180ae7bef5f67589281e8deb87c02/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e5ae57760ae92b8d7913bd9a0dfb6ffb7ad5d42381840cecba055cd0c3a09ab9/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/64db0127cb43976a48a2495ae1b3af922043268f43bd1ef362142575f9aba7b7/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/4a317fbd993a8e560ea1e4fd034b9c6909c41889d8048d20ecefd90954e479e6/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d7008a6f12d3bbf082c9b78c7b917e3d104d29b0ea825394842d37b09282abee/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/68f697d0c71660a69be944c0e50bec8d7686cb70945e805c1524d9196a46ee2d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ba548468b6fe358acdddf8e8ae9e84f3ecb75d44021c15b42e3b6be88c9a7276/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/04cbe694976300f66c6c1d5716dece00ef8440e0ff15f7a970d7a25b9b4a6ce8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b334149a4112af6e04070fefc686a5532f221e071d794e76e84a02f5aa717539/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4c24b4e1715dfa1bb32f063bfd14d1bd03260dc8ad5a4b020b84e2671e990f4a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/14701166099d5e5bf388be9207606881f3781846d6671c4e18bfe7e97c8de6f5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/00d4158de6d92fecf4b93f410842b453b0f934cc102bcb4e5a7d32d4a4914fb4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/82d16d33132d061ef98eca32f845c5a0056c93cae722256b5273e262d1836c0c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/94c5d4de3adbea8964ba62477c10e0f5a5f614f26e263a34c95245256f2bb18d/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/39b77fac7332d2b3f3db1a1492f4b20d7ab89ff2c829055caa0493e9ebd3ca64/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a87e7e7b3cfa15b983a263a07ecad5e763c6f2ef6fda86b04064fdd8dd4ac236/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/621b1b09e916b3fd7b69b67322e0d5f8bb971ed54e4b2a973666fa09e15345cd/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/de0ec47476462bb7fec1442731fb3371d2044c524b0021f82c04e36dcc550529/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6bfa5c0d42b40153ecde7a9a6ec5a04eb2b33d58d53e9565b4ee3c7dda45de08/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b93c4ac814fb476a98d3696a250b15fcbec8d95307701eeadba3cde5606ab302/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/9e4daddd0513630af46f408a559aace42b409bc06c804d81b1ae3bba80611046/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/a300432377c360082ba50183a574a372797baa1587f8e74d252b3a4f5285d6c6/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5d3a35694d84b50c73863db3dd3ea60b0985401e020fc7707c9a42f8ddbb1ac5/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/79f89c761f13775875acf3e05277d6fb79e5c1f843d01599d070324c53dc3876/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0019a59d756d2d9bd329f50ebdb91abe7088a3a4261d066424e4bc8c2e8f0e27/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/f0ea757fb239b9595aed0f6120392c866310961f79005d934d50bfbe42b7ae89/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/9adb2c5e2a0a9e136b25f0015db29980a5b97aef179a34501c98c167764bbfe8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ac5fc8c4939002ab912e17728611d09b66b046e33cc137146cacba5896e0fefe/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/9b58c381970daf0c6ebc6fad0ba0602833aef8664eaddc0a367bd10490f1f07f/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/448c8b90868e820cc912eda1f0a4e0d6f1a8da7068f3f8406990e866511d4467/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/b6ecdd6e53f7173d59614da53b3e9abf59cf517b525be77d0ade04cdc2328ee5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/5dacce3f4bfd40e4a56ed99aa100d735bb6b863fe79eab5429571ddfcb87cb5c/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/5bd83075013736ec504ca1f1af38b2b03cd9b2c51fac248b3863d29d607a676f/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/1d44e2097a904a6bbf3e207d72d58354392c3c280cf2a303b28a83895818a798/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a3dac96389020e9202ee19d60910090c6b42552bd214d90a2f55774190512758/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/7b4f2ac5091bc4ba8f55651dff0174ffeeef82ec13c2f05a28bc4a2542553f03/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/57e73ada1e8027eaaf5786326eea68909a3d68665f66287728951a30d7faa528/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/aa3d430a7c2a91c684243128f55e3d993e32b2b4b892da2454ff6c7c4fceebf2/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/5072cc6fb2ce1a6b6477a1e04b6b71d1f0abe58fdf97f261b07e6b20505a6151/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/d65ca70e8bcb18525dbc915b45d2c8218743231112a015e6f2ebb0360f755c9b/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/ce99c20c5b320269c52690ea5b1be64f15e4d3020582a879d7ad1ae6e6068ffe/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/3cc28ceaa098338826154f166b8c03bdcc428e1ef309622903c7125bf960955f/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/789be57eb5100201f42c0d6729142a236ac5944f7476fca114008633cc726ca2/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4c1bdb6d11cfc19e71e6488d2cebab28faebb853d23a6730473441b589b59450/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b3df3c4340c4ac125294669ff12645ba7eb920a035c9a2ceae44a88e44495e83/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c8114dfc1a2c05cb19e8cf9805b4e245eb8c8ca0cf4a38ed7b367e2445165f52/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2f0a26538aac93f3fdb304d124572b48c04a8dde33ca7a9b18411d287fa6fa03/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/ac1b154fe6ea05fd10028eb095ab479e7253f8851337dc570a3c66daa6d7d843/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/16565cdf40302d2943138a66615132b115dc5d49e6dc91a808087516a98242ab/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/bd3a2aa156ba76a0a570b927273b0758acd41b1488ef510224f3834e5e06f77e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ecd2bb187538a35dc1758db2b6b8ea9c1cd8a218241b4dc3bb936cb1d2f11db4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/52b09342d4dd3413cd8a4d5cc088ac0f554f5d6ea332cd2ea87faf854b68c5f3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c502e97ee0141f17d2b994bee1787cabae94beceefb9d2a226b218cd5ec12b82/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 21:34:12.922070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.921705 5091 manager.go:217] Machine: {Timestamp:2026-11-02 21:34:12.921236278 +0000 UTC m=+0.083331389 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102280 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b46141ec9da8951268bb0bf599ec6 SystemUUID:ec2b4614-1ec9-da89-5126-8bb0bf599ec6 BootID:43bd0896-3e71-4a74-8bce-20186123f69f Filesystems:[{Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff3812dd-9c0e-482d-b760-4540712ff854/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82eb2931-ff21-4bf4-9589-995055842cb4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5af35ed2-0ae8-4679-9c23-419d9a64a2e2/volumes/kubernetes.io~projected/kube-api-access-zcdh9 DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d07fa63e1362d27ca04c030366fe0ed31872fe8dcc3cea399bd4a3a4e85c38e4/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f0291ae-a1d6-4312-9e07-da45a493466e/volumes/kubernetes.io~projected/kube-api-access-dp982 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea72a1f38cd85fda3c29b74252ed5c0def91ddbe283995cb18fb661b049b8099/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06076e5c-0467-47fa-a47b-946e170df5bb/volumes/kubernetes.io~projected/kube-api-access-xh29q DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f6b3b2e-5e40-4067-9e29-a891a598286f/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1404835-f6fb-400f-b107-476157b43c7d/volumes/kubernetes.io~projected/kube-api-access-fkl8p DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f6b3b2e-5e40-4067-9e29-a891a598286f/volumes/kubernetes.io~projected/kube-api-access-2pztg DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb533acd-73cb-4f30-a60e-e96debb69f25/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c50b3dbd3d542d1cacfec503056458ced5515181138c4850c93c10b221540df/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fecff0202ecf96596599802f0bdee29c3d5be3068af9c25551cbce72f5a56b5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc6dbf33-7424-4695-9e9a-6a99475477dd/volumes/kubernetes.io~projected/kube-api-access-5jzpg DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bf51ab4cb95e320afc370ca86ff1e778600ecc5a04ce3d896e62bac39fc6fb4/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8deefde1a35288ad85885d426a8e8e5c08baf8fbabbb418fac88d2f9d911eceb/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3900b59-124a-4dc5-9cb4-e5e98253eb5a/volumes/kubernetes.io~projected/kube-api-access-rp8kt DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6/volumes/kubernetes.io~projected/kube-api-access-576ks DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7575b0455e87922523a01265f07a1ce8c048d825447762dc30d65981f469f9e7/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c/volumes/kubernetes.io~projected/kube-api-access-4vz8v DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/324b3317364b7cc9bc667ca5fd8249954f40bbe415460bb6aa1c9b29dd35b10f/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e1aacabbc767a44075e0c316f4ef8e9f8a518e2bb05e1b2762c438bbd377b27/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b/volumes/kubernetes.io~projected/kube-api-access-mkcdz DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4016e9e3-e3e8-454a-a635-49970015c350/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a045057efad7882150d785c18318741d59182d2061f3295515330787149797d5/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a2b13a-685e-4ba3-8370-e059d7fe5e26/volumes/kubernetes.io~projected/kube-api-access-27524 DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~projected/kube-api-access-n78t6 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc6dbf33-7424-4695-9e9a-6a99475477dd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b492161-d0aa-42e5-8947-9403654941e3/volumes/kubernetes.io~projected/kube-api-access-rzxvm DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ccc649fe0c520be2c6ebeafc895ba76ccf466049373bc58c9df28909c1f5d5a/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e248ed2629aebc6f32522606ba54e98ea454668a3278e8b5313ccb4d65cf4133/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/328ffd7a1f163d37332da6d1f056d18242cb6f05ce142bcde091d0f6c1d75504/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4012a1c20ce04e2fc39c0d57952fe67490e8a46d5c5a5495a60f64b92019049/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/979b4210b451cbf43712ba008c2bbd99f5a31e552db53ff8405c00c8d0627133/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/367a2050-9d6e-4486-af6c-1f19921cb8cf/volumes/kubernetes.io~projected/kube-api-access-qk4xg DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ef9cdd0069a19e740a5ce1a08c2035bc155c3f5830179d600d5ad3f1aaff21b/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24c005d3-f7c3-4292-ac2d-ff26ec4065fd/volumes/kubernetes.io~projected/kube-api-access-rmtm7 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/481f04d7-a2a2-472c-8a0b-744c1ca0e486/volumes/kubernetes.io~projected/kube-api-access-z8vvn DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b9bee42d0adf5c37d868eab7aed69ba59a09ff9514cf10d892597becc5abda6/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/350e88f761e0273eba167b3cb4beb882db167ea77b0ed35c0202a3f65830cf8e/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0a36a5-f22c-4759-8e09-724db23b7c04/volumes/kubernetes.io~projected/kube-api-access-8s6pr DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe41410a-c20e-48c0-b3c3-f531c79c3fe5/volumes/kubernetes.io~projected/kube-api-access-mlqts DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31e0fa56-e920-4006-91c5-7425cd741ead/volumes/kubernetes.io~projected/kube-api-access-xrf85 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82eb2931-ff21-4bf4-9589-995055842cb4/volumes/kubernetes.io~projected/kube-api-access-5mv7c DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes/kubernetes.io~projected/kube-api-access-ghzsm DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/volumes/kubernetes.io~projected/kube-api-access-zk7kk DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9aa8f73d6fdc596e9b08b5675a87750877f5140a78fef7bf70ea2f834b149aac/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eac1a0f2d6f3c0da853202764c38a016b376864df88d1b647ec8690a7585501d/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3c60cbfd449ab5c7a2b35e14ec08aa3fa3a78c10627898556ef255ca84a7ca6/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e987d331-5974-430e-8a6b-d1b1dcbb7fe9/volumes/kubernetes.io~projected/kube-api-access-bffx9 DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a/volumes/kubernetes.io~projected/kube-api-access-rnnd2 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16b4573f7e58e9753180a87c59f2a73b596852099323f3b4163a6e26fb0c18db/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3/volumes/kubernetes.io~projected/kube-api-access-fw7rx DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7b492161-d0aa-42e5-8947-9403654941e3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff3812dd-9c0e-482d-b760-4540712ff854/volumes/kubernetes.io~projected/kube-api-access-nn9vg DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes/kubernetes.io~projected/kube-api-access-22hsk DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e/volumes/kubernetes.io~projected/kube-api-access-c5dgc DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d49fbb10561d511b3f0d6c954119f8c776741333e25e9c42da5d51781b31b06/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d020426a-2902-454c-8b86-7ecc7c666826/volumes/kubernetes.io~projected/kube-api-access-qm6dc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99921fd6-6911-4ac5-852d-e90337681ce9/volumes/kubernetes.io~projected/kube-api-access-lc2h7 DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cf09ffe-d458-41e7-b9f4-70bf537e64b0/volumes/kubernetes.io~projected/kube-api-access-tmfmr DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2ef9cdd0069a19e MacAddress:c6:c1:23:78:44:ac Speed:10000 Mtu:8901} {Name:31d8e9ff0ab355c MacAddress:e2:10:4e:44:8e:7b Speed:10000 Mtu:8901} {Name:328ffd7a1f163d3 MacAddress:fa:48:50:91:82:96 Speed:10000 Mtu:8901} {Name:350e88f761e0273 MacAddress:ba:bd:72:5c:6e:30 Speed:10000 Mtu:8901} {Name:3ccc649fe0c520b MacAddress:9a:ea:96:fd:4f:35 Speed:10000 Mtu:8901} {Name:6bf51ab4cb95e32 MacAddress:9a:30:3f:d1:ac:dc Speed:10000 Mtu:8901} {Name:7575b0455e87922 MacAddress:6e:ad:c7:20:13:24 Speed:10000 Mtu:8901} {Name:8deefde1a35288a MacAddress:1a:1d:45:60:ec:24 Speed:10000 Mtu:8901} {Name:979b4210b451cbf MacAddress:c6:23:fb:34:3e:0e Speed:10000 Mtu:8901} {Name:9aa8f73d6fdc596 MacAddress:86:4e:08:29:ce:11 Speed:10000 Mtu:8901} {Name:9b9bee42d0adf5c MacAddress:72:34:c1:52:9b:0a Speed:10000 Mtu:8901} {Name:a045057efad7882 MacAddress:c2:b5:55:80:ef:8a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:69:a0:4d:f2:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:b5:41:ea:36:65 Speed:0 Mtu:8901} {Name:e248ed2629aebc6 MacAddress:ea:07:c3:17:8e:e6 Speed:10000 Mtu:8901} {Name:ea72a1f38cd85fd MacAddress:1e:ba:12:54:39:50 Speed:10000 Mtu:8901} {Name:eac1a0f2d6f3c0d MacAddress:5e:63:63:08:66:bd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:69:a0:4d:f2:f1 Speed:0 Mtu:9001} {Name:f3c60cbfd449ab5 MacAddress:f2:05:2b:0b:f6:7d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:18:1d:9f:32:b7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:e4:ad:cb:40:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:34:12.922360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922073 5091 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:34:12.922360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922152 5091 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:34:12.922450 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922419 5091 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:34:12.922761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922440 5091 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-36.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:34:12.922830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922762 5091 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:34:12.922830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922770 5091 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:34:12.922830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922781 5091 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:34:12.922830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922793 5091 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:34:12.923000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.922887 5091 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:34:12.923051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923041 5091 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:34:12.923146 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923138 5091 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:34:12.923194 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923170 5091 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:34:12.923194 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923180 5091 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:34:12.923194 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923189 5091 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:34:12.923321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923197 5091 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:34:12.923321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.923205 5091 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:34:12.924091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.924078 5091 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:34:12.924091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.924091 5091 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:34:12.924549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.924531 5091 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:34:12.927721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.927703 5091 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:34:12.927721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.927720 5091 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:34:12.927844 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.927824 5091 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:34:12.928076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928063 5091 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:34:12.928121 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928111 5091 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:34:12.928226 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928215 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:34:12.928265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928234 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:34:12.928265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928243 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:34:12.928265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928251 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:34:12.928265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928260 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928270 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928285 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928294 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928305 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928315 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928333 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:34:12.928367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928346 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:34:12.928657 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928376 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:34:12.928657 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.928384 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:34:12.932695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.932516 5091 server.go:1263] "Started kubelet" Nov 02 21:34:12.935702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.932891 5091 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:34:12.935702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.932970 5091 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:34:12.935702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.933071 5091 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:34:12.935702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.933416 5091 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:34:12.935702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.934485 5091 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:34:12.933069 ip-10-0-128-36 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.937921 5091 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.937938 5091 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.938056 5091 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:28:33 +0000 UTC" deadline="2028-06-10 15:01:05.264276231 +0000 UTC" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.938079 5091 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.938091 5091 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.938088 5091 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14057h26m52.326191511s" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:12.938147 5091 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:34:12.939185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.938926 5091 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:34:12.945043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.943876 5091 factory.go:55] Registering systemd factory Nov 02 21:34:12.945043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.943898 5091 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:34:12.945236 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945156 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99921fd6-6911-4ac5-852d-e90337681ce9" volumeName="kubernetes.io/projected/99921fd6-6911-4ac5-852d-e90337681ce9-kube-api-access-lc2h7" seLinuxMountContext="" Nov 02 21:34:12.945236 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945204 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c" volumeName="kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cni-binary-copy" seLinuxMountContext="" Nov 02 21:34:12.945236 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945216 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c" volumeName="kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-daemon-config" seLinuxMountContext="" Nov 02 21:34:12.945236 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945228 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a2b13a-685e-4ba3-8370-e059d7fe5e26" volumeName="kubernetes.io/projected/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-api-access-27524" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945239 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1404835-f6fb-400f-b107-476157b43c7d" volumeName="kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945250 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" volumeName="kubernetes.io/projected/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-kube-api-access-fw7rx" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945260 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492161-d0aa-42e5-8947-9403654941e3" volumeName="kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-snapshots" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945271 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492161-d0aa-42e5-8947-9403654941e3" volumeName="kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-tmp" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945284 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6dbf33-7424-4695-9e9a-6a99475477dd" volumeName="kubernetes.io/secret/fc6dbf33-7424-4695-9e9a-6a99475477dd-serving-cert" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945295 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945308 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0a36a5-f22c-4759-8e09-724db23b7c04" volumeName="kubernetes.io/projected/0d0a36a5-f22c-4759-8e09-724db23b7c04-kube-api-access-8s6pr" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945318 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a05d0e-bf88-42c3-a8d7-f62e1b039a0f" volumeName="kubernetes.io/projected/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-kube-api-access-zk7kk" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945329 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06076e5c-0467-47fa-a47b-946e170df5bb" volumeName="kubernetes.io/projected/06076e5c-0467-47fa-a47b-946e170df5bb-kube-api-access-xh29q" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945341 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d020426a-2902-454c-8b86-7ecc7c666826" volumeName="kubernetes.io/projected/d020426a-2902-454c-8b86-7ecc7c666826-kube-api-access-qm6dc" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945354 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0291ae-a1d6-4312-9e07-da45a493466e" volumeName="kubernetes.io/projected/0f0291ae-a1d6-4312-9e07-da45a493466e-kube-api-access-dp982" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945365 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557fdf7f-9d7a-4de7-9819-becbc1fbfb8a" volumeName="kubernetes.io/projected/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-kube-api-access-rnnd2" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945377 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6466fd-6cf7-4d27-bd08-4eaf51c5990e" volumeName="kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-config" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945388 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945398 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945409 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a05d0e-bf88-42c3-a8d7-f62e1b039a0f" volumeName="kubernetes.io/secret/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-serving-cert" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945420 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4016e9e3-e3e8-454a-a635-49970015c350" volumeName="kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945431 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481f04d7-a2a2-472c-8a0b-744c1ca0e486" volumeName="kubernetes.io/configmap/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-config" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945442 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6466fd-6cf7-4d27-bd08-4eaf51c5990e" volumeName="kubernetes.io/secret/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945460 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6" volumeName="kubernetes.io/projected/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-kube-api-access-576ks" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945471 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492161-d0aa-42e5-8947-9403654941e3" volumeName="kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-trusted-ca-bundle" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945482 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e987d331-5974-430e-8a6b-d1b1dcbb7fe9" volumeName="kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9" seLinuxMountContext="" Nov 02 21:34:12.945480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945493 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc532d99-0b1a-40fb-ad7c-42c4cfbb879e" volumeName="kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-tmp" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945504 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f6b3b2e-5e40-4067-9e29-a891a598286f" volumeName="kubernetes.io/configmap/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-cabundle" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945527 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6" volumeName="kubernetes.io/empty-dir/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-tmp-dir" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945539 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492161-d0aa-42e5-8947-9403654941e3" volumeName="kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-service-ca-bundle" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945550 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf09ffe-d458-41e7-b9f4-70bf537e64b0" volumeName="kubernetes.io/projected/7cf09ffe-d458-41e7-b9f4-70bf537e64b0-kube-api-access-tmfmr" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945560 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb533acd-73cb-4f30-a60e-e96debb69f25" volumeName="kubernetes.io/configmap/eb533acd-73cb-4f30-a60e-e96debb69f25-konnectivity-ca" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945570 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb533acd-73cb-4f30-a60e-e96debb69f25" volumeName="kubernetes.io/secret/eb533acd-73cb-4f30-a60e-e96debb69f25-agent-certs" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945581 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1404835-f6fb-400f-b107-476157b43c7d" volumeName="kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945592 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945602 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f6b3b2e-5e40-4067-9e29-a891a598286f" volumeName="kubernetes.io/projected/0f6b3b2e-5e40-4067-9e29-a891a598286f-kube-api-access-2pztg" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945613 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c005d3-f7c3-4292-ac2d-ff26ec4065fd" volumeName="kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-default-certificate" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945624 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e0fa56-e920-4006-91c5-7425cd741ead" volumeName="kubernetes.io/projected/31e0fa56-e920-4006-91c5-7425cd741ead-kube-api-access-xrf85" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945635 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367a2050-9d6e-4486-af6c-1f19921cb8cf" volumeName="kubernetes.io/projected/367a2050-9d6e-4486-af6c-1f19921cb8cf-kube-api-access-qk4xg" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945646 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" volumeName="kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-binary-copy" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945657 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" volumeName="kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945682 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d020426a-2902-454c-8b86-7ecc7c666826" volumeName="kubernetes.io/configmap/d020426a-2902-454c-8b86-7ecc7c666826-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945693 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1404835-f6fb-400f-b107-476157b43c7d" volumeName="kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945703 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492161-d0aa-42e5-8947-9403654941e3" volumeName="kubernetes.io/projected/7b492161-d0aa-42e5-8947-9403654941e3-kube-api-access-rzxvm" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945715 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945725 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945737 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945748 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af35ed2-0ae8-4679-9c23-419d9a64a2e2" volumeName="kubernetes.io/projected/5af35ed2-0ae8-4679-9c23-419d9a64a2e2-kube-api-access-zcdh9" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945758 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6466fd-6cf7-4d27-bd08-4eaf51c5990e" volumeName="kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945768 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6466fd-6cf7-4d27-bd08-4eaf51c5990e" volumeName="kubernetes.io/projected/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-kube-api-access-c5dgc" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945779 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492161-d0aa-42e5-8947-9403654941e3" volumeName="kubernetes.io/secret/7b492161-d0aa-42e5-8947-9403654941e3-serving-cert" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945789 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b" volumeName="kubernetes.io/projected/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b-kube-api-access-mkcdz" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945800 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc532d99-0b1a-40fb-ad7c-42c4cfbb879e" volumeName="kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-tuned" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945811 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6dbf33-7424-4695-9e9a-6a99475477dd" volumeName="kubernetes.io/configmap/fc6dbf33-7424-4695-9e9a-6a99475477dd-config" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945823 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6dbf33-7424-4695-9e9a-6a99475477dd" volumeName="kubernetes.io/projected/fc6dbf33-7424-4695-9e9a-6a99475477dd-kube-api-access-5jzpg" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945834 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945852 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3900b59-124a-4dc5-9cb4-e5e98253eb5a" volumeName="kubernetes.io/projected/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-kube-api-access-rp8kt" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945865 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1c67a3a-2cab-4583-976c-a23c07673db0" volumeName="kubernetes.io/configmap/c1c67a3a-2cab-4583-976c-a23c07673db0-nginx-conf" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945876 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945887 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945897 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe41410a-c20e-48c0-b3c3-f531c79c3fe5" volumeName="kubernetes.io/projected/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kube-api-access-mlqts" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945908 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff3812dd-9c0e-482d-b760-4540712ff854" volumeName="kubernetes.io/secret/ff3812dd-9c0e-482d-b760-4540712ff854-cert" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945919 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c6466fd-6cf7-4d27-bd08-4eaf51c5990e" volumeName="kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-env-overrides" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945930 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82eb2931-ff21-4bf4-9589-995055842cb4" volumeName="kubernetes.io/projected/82eb2931-ff21-4bf4-9589-995055842cb4-kube-api-access-5mv7c" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945940 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82eb2931-ff21-4bf4-9589-995055842cb4" volumeName="kubernetes.io/secret/82eb2931-ff21-4bf4-9589-995055842cb4-serving-cert" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945951 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945961 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc532d99-0b1a-40fb-ad7c-42c4cfbb879e" volumeName="kubernetes.io/projected/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-kube-api-access-n78t6" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945971 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f6b3b2e-5e40-4067-9e29-a891a598286f" volumeName="kubernetes.io/secret/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-key" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945982 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" volumeName="kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.945993 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6" volumeName="kubernetes.io/configmap/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-config-volume" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946005 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a05d0e-bf88-42c3-a8d7-f62e1b039a0f" volumeName="kubernetes.io/configmap/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-config" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946015 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc9f5580-bb4a-4c27-8672-949c0507a719" volumeName="kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946025 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946037 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c005d3-f7c3-4292-ac2d-ff26ec4065fd" volumeName="kubernetes.io/projected/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-kube-api-access-rmtm7" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946048 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c005d3-f7c3-4292-ac2d-ff26ec4065fd" volumeName="kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-stats-auth" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946059 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82eb2931-ff21-4bf4-9589-995055842cb4" volumeName="kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-config" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946070 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c425440-6797-483a-b758-10dcc09a32c7" volumeName="kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946081 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0a36a5-f22c-4759-8e09-724db23b7c04" volumeName="kubernetes.io/configmap/0d0a36a5-f22c-4759-8e09-724db23b7c04-serviceca" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946091 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0291ae-a1d6-4312-9e07-da45a493466e" volumeName="kubernetes.io/configmap/0f0291ae-a1d6-4312-9e07-da45a493466e-telemetry-config" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946102 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82eb2931-ff21-4bf4-9589-995055842cb4" volumeName="kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-trusted-ca" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946113 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff3812dd-9c0e-482d-b760-4540712ff854" volumeName="kubernetes.io/projected/ff3812dd-9c0e-482d-b760-4540712ff854-kube-api-access-nn9vg" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946123 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557fdf7f-9d7a-4de7-9819-becbc1fbfb8a" volumeName="kubernetes.io/empty-dir/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-tmp-dir" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946134 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c" volumeName="kubernetes.io/projected/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-kube-api-access-4vz8v" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946146 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a2b13a-685e-4ba3-8370-e059d7fe5e26" volumeName="kubernetes.io/empty-dir/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-data-volume" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946158 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1404835-f6fb-400f-b107-476157b43c7d" volumeName="kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946169 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481f04d7-a2a2-472c-8a0b-744c1ca0e486" volumeName="kubernetes.io/projected/481f04d7-a2a2-472c-8a0b-744c1ca0e486-kube-api-access-z8vvn" seLinuxMountContext="" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946181 5091 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:34:12.946511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.946189 5091 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:34:12.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.947454 5091 factory.go:153] Registering CRI-O factory Nov 02 21:34:12.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.947472 5091 factory.go:223] Registration of the crio container factory successfully Nov 02 21:34:12.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.947538 5091 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:34:12.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.947557 5091 factory.go:103] Registering Raw factory Nov 02 21:34:12.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.947570 5091 manager.go:1199] Started watching for new ooms in manager Nov 02 21:34:12.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.947923 5091 manager.go:319] Starting recovery of all containers Nov 02 21:34:12.965704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.958811 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/1.log" Nov 02 21:34:12.965704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.963039 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/1.log" Nov 02 21:34:12.967270 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.967223 5091 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:34:12.968590 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.968561 5091 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:34:12.968590 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.968589 5091 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:34:12.968780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.968613 5091 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:34:12.968780 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:12.968658 5091 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:34:12.983794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.983758 5091 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="cb9d9a07343438e1a335212d786286bc164ea9b4e4dc10119aa3fffe4a623d39" exitCode=0 Nov 02 21:34:12.983794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.983784 5091 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="aaedde40f329f6cf0dffcdad69a868c42baa9c4b1b835400b525f689626d776e" exitCode=0 Nov 02 21:34:12.983926 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.983794 5091 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="9eab52650e1633f8312be61e0b48071862a5deccb03773b0e126b323a8cc221b" exitCode=0 Nov 02 21:34:12.983926 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.983815 5091 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="c41b6094c1946d698129bd356c9844421aeb845cbd5131fc7e119171175dbe8e" exitCode=0 Nov 02 21:34:12.983926 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.983822 5091 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="52fa4050783a17c6c11f8c7418c350450a913e1c75b8bfa11c8d571e0badcf9a" exitCode=0 Nov 02 21:34:12.983926 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:12.983829 5091 generic.go:356] "Generic (PLEG): container finished" podID="56d0b0db-ceaa-49c5-bff5-16b1d7c370e3" containerID="dc0f427c24582563d16460de302abecf88bc96966307a92e9700148295e7aaaf" exitCode=0 Nov 02 21:34:13.016039 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.016017 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/1.log" Nov 02 21:34:13.016154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.016045 5091 generic.go:356] "Generic (PLEG): container finished" podID="82eb2931-ff21-4bf4-9589-995055842cb4" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" exitCode=255 Nov 02 21:34:13.030960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.030894 5091 generic.go:356] "Generic (PLEG): container finished" podID="9d6ec7fa97b5465128bc132a816f76dd" containerID="4c35409c103b018a01438eddff6081d0316600f898c6bec7a4a1ac9d67b5af5c" exitCode=0 Nov 02 21:34:13.038235 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.038213 5091 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:34:13.048644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.048627 5091 manager.go:324] Recovery completed Nov 02 21:34:13.068366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.068347 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:13.069162 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.069144 5091 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 21:34:13.069382 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.069368 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:13.069421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.069396 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:13.069421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.069406 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:13.071357 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071338 5091 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:34:13.071357 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071353 5091 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:34:13.071484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071367 5091 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:34:13.071484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071344 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:13.071484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071472 5091 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:34:13.071586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071483 5091 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:34:13.071586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071500 5091 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:34:13.071586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071505 5091 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:34:13.071586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071510 5091 policy_none.go:50] "Start" Nov 02 21:34:13.071586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071530 5091 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:34:13.071586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071554 5091 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:34:13.071862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071696 5091 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:34:13.071862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071704 5091 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:34:13.071862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.071710 5091 policy_none.go:44] "Start" Nov 02 21:34:13.072209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.072194 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:13.072265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.072220 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:13.072265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.072230 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:13.075896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.075869 5091 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:34:13.081021 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081008 5091 manager.go:341] "Starting Device Plugin manager" Nov 02 21:34:13.081098 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.081032 5091 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:34:13.081098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081041 5091 server.go:87] "Starting device plugin registration server" Nov 02 21:34:13.081296 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081282 5091 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:34:13.081342 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081299 5091 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:34:13.081387 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081372 5091 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:34:13.081499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081463 5091 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:34:13.081549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081507 5091 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:34:13.081549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081515 5091 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:34:13.081632 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.081563 5091 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:34:13.081497558Z","UUID":"6bfbaf93-ebab-4ace-9b37-31479fb60f56","Handler":null,"Name":"","Endpoint":""} Nov 02 21:34:13.083155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.083137 5091 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:34:13.083155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.083158 5091 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:34:13.083337 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.083316 5091 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:34:13.083378 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.083357 5091 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-36.us-west-1.compute.internal\" not found" Nov 02 21:34:13.132046 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.132011 5091 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:13.132182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.132140 5091 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:34:13.179904 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.179880 5091 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:13.179994 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.179913 5091 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:13.181486 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.181471 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:13.182062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.182050 5091 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:13.183709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.183693 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:13.183779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.183719 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:13.183779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.183734 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-36.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:13.183779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.183776 5091 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.269547 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.269502 5091 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal"] Nov 02 21:34:13.270157 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270141 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.270157 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270136 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"356429dd5260af9a132a468067041120","Type":"ContainerStarted","Data":"b9c5923ada059b2985b256c9a23d6ba32fa7854ebf2707e48c654d198e5511da"} Nov 02 21:34:13.270291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270171 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"356429dd5260af9a132a468067041120","Type":"ContainerStarted","Data":"e0dd5efac54ec296542e83c0d560ab9273241e645d11a3ad7fcc5dc0bb6b257e"} Nov 02 21:34:13.270291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270207 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"9d6ec7fa97b5465128bc132a816f76dd","Type":"ContainerStarted","Data":"02ad86fdd1565226d09dbf1f986f96ff801061fd5f2516f7bd5a655704cbbe6c"} Nov 02 21:34:13.270291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270219 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"9d6ec7fa97b5465128bc132a816f76dd","Type":"ContainerDied","Data":"4c35409c103b018a01438eddff6081d0316600f898c6bec7a4a1ac9d67b5af5c"} Nov 02 21:34:13.270291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270231 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" event={"ID":"9d6ec7fa97b5465128bc132a816f76dd","Type":"ContainerStarted","Data":"5b0944008539e46076258fcedee971944fcc1af0adf24117530b0c9bbf2f6c9d"} Nov 02 21:34:13.270437 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.270365 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.298554 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.298505 5091 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.298619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.298562 5091 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.343371 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.343349 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:13.343449 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.343390 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.352922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.352899 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:13.353007 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:13.352938 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.434492 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.434469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356429dd5260af9a132a468067041120-config\") pod \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"356429dd5260af9a132a468067041120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.434586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.434495 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.434586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.434512 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.535349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.535322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356429dd5260af9a132a468067041120-config\") pod \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"356429dd5260af9a132a468067041120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.535349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.535351 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.535502 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.535370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.535502 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.535421 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.535502 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.535429 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356429dd5260af9a132a468067041120-config\") pod \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"356429dd5260af9a132a468067041120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.535502 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.535432 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6ec7fa97b5465128bc132a816f76dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" (UID: \"9d6ec7fa97b5465128bc132a816f76dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:13.923882 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:13.923851 5091 apiserver.go:52] "Watching apiserver" Nov 02 21:34:14.036959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.036936 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:14.037096 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.037074 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:14.076777 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.076756 5091 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:34:14.077611 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.077585 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-sr2gb","kube-system/kas-connection-checker-685895c7c9-nkwv9","openshift-insights/insights-runtime-extractor-b5tqm","openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l","openshift-cluster-node-tuning-operator/tuned-jxnx6","openshift-image-registry/node-ca-9njpz","kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j","openshift-multus/multus-cbj8m","openshift-service-ca/service-ca-5fffc97bdf-tzk7v","openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h","openshift-image-registry/image-registry-657747857b-smdws","openshift-ovn-kubernetes/ovnkube-node-cswj4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47","openshift-dns/dns-default-vn9dz","openshift-network-operator/iptables-alerter-2nwsl","openshift-image-registry/image-registry-554b69489-rlbg8","openshift-insights/insights-operator-7c4986d449-nj4zc","openshift-console-operator/console-operator-7cd56b8dbc-kklpk","openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9","openshift-insights/periodic-gathering-w59fz-jvbfg","openshift-ingress/router-default-cd7d6d4d5-p5mwz","openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq","openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw","openshift-network-diagnostics/network-check-target-9xtbf","openshift-ingress-canary/ingress-canary-hsbjx","kube-system/kas-connection-checker-685895c7c9-xxjw8","kube-system/kas-connection-checker-685895c7c9-lzqpt","openshift-network-console/networking-console-plugin-86c478bb88-ptr6r","openshift-multus/multus-additional-cni-plugins-w9h49","openshift-multus/network-metrics-daemon-tfzjr","openshift-dns/node-resolver-4qplz","kube-system/konnectivity-agent-wk8sg","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h"] Nov 02 21:34:14.079530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.079506 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.082554 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.082534 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.083958 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.083940 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.084349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.084329 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.085216 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.085200 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:14.087126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.087110 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:14.091936 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.091919 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.093083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.093063 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.094344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.094327 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.095323 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.095305 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:14.095377 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.095358 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:14.095728 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.095710 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:14.095840 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.095752 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-36.us-west-1.compute.internal" Nov 02 21:34:14.131569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.131546 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.131660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.131647 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.131729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.131690 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.131922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.131903 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 21:34:14.131922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.131921 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.132087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.131970 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 21:34:14.132087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.132041 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.132087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.132082 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 21:34:14.132221 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.132098 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.132864 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.132836 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-92ldj\"" Nov 02 21:34:14.132954 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.132938 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pwlbm\"" Nov 02 21:34:14.133841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.133814 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 21:34:14.136857 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.136837 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 21:34:14.138161 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138121 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 21:34:14.138623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138549 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 21:34:14.138623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138580 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 21:34:14.138623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138589 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 21:34:14.138623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138620 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138650 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138698 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jfs7n\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138725 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rmsz2\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138738 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138737 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138770 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138773 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138789 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:34:14.138877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138833 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2lcwh\"" Nov 02 21:34:14.139108 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.138941 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.140661 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.140646 5091 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:34:14.182233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182162 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.182233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182166 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.182233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182224 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182234 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182227 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182238 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182275 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182295 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182302 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182344 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.182394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182394 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5wddx\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182394 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s5wdb\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182454 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182478 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zd85x\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182534 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x9kkw\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182553 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182574 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 21:34:14.182737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182681 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182796 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsxz6\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182810 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182858 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182860 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182874 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182913 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 21:34:14.183033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.182998 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mb8sd\"" Nov 02 21:34:14.183254 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183180 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2mch2\"" Nov 02 21:34:14.183254 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183245 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.183355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183268 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 21:34:14.183355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183283 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 21:34:14.183355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183295 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:34:14.183355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183297 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tjx6v\"" Nov 02 21:34:14.183355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183300 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183359 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183333 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183360 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183374 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183371 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183421 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qbt7b\"" Nov 02 21:34:14.183540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183533 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.183817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183632 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.183817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183743 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 21:34:14.183817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183752 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.183817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183784 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:34:14.184004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183831 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.184004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183867 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:34:14.184004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183895 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7tsg5\"" Nov 02 21:34:14.184004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183908 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.184004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.183977 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:34:14.184237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184014 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nr5z4\"" Nov 02 21:34:14.184237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184095 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 21:34:14.184237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184093 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.184237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184118 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:34:14.184237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184175 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:34:14.184237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184204 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184249 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f7c29\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184253 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184278 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184328 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7clxp\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184337 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184331 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184333 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-stcz5\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184368 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fbz7r\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184384 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184385 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184419 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 21:34:14.184460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184419 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:34:14.184847 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.184741 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 21:34:14.186556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186541 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rwnh\"" Nov 02 21:34:14.186615 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186567 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b8lsz\"" Nov 02 21:34:14.186615 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186586 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jhcsj\"" Nov 02 21:34:14.186749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186622 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:34:14.186749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186643 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.186749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186643 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.186749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186706 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186749 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186757 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zfnt5\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186751 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186771 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186782 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plzkt\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186782 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186787 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vkfz5\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186813 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186834 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95jdp\"" Nov 02 21:34:14.186899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186895 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sb4td\"" Nov 02 21:34:14.187163 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186949 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:14.187163 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.186986 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:34:14.238827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.238806 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7rx\" (UniqueName: \"kubernetes.io/projected/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-kube-api-access-fw7rx\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.238916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.238831 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.238916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.238849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27524\" (UniqueName: \"kubernetes.io/projected/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-api-access-27524\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.238916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.238865 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6dbf33-7424-4695-9e9a-6a99475477dd-serving-cert\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:14.239084 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.238959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:14.239084 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.238989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.239084 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239017 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cni-binary-copy\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.239084 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc6dbf33-7424-4695-9e9a-6a99475477dd-serving-cert\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:14.239084 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239067 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f0291ae-a1d6-4312-9e07-da45a493466e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.239308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:14.239308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239251 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-config-volume\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.239308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239278 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb533acd-73cb-4f30-a60e-e96debb69f25-agent-certs\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:34:14.239308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239302 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp982\" (UniqueName: \"kubernetes.io/projected/0f0291ae-a1d6-4312-9e07-da45a493466e-kube-api-access-dp982\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239351 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-hosts-file\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d0a36a5-f22c-4759-8e09-724db23b7c04-host\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239404 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d020426a-2902-454c-8b86-7ecc7c666826-iptables-alerter-script\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239406 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cni-binary-copy\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239403 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0f0291ae-a1d6-4312-9e07-da45a493466e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239456 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.239494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239492 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-config-volume\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239508 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-tmp-dir\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239539 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb533acd-73cb-4f30-a60e-e96debb69f25-konnectivity-ca\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-systemd-units\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239594 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-systemd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239619 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239664 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-var-lib-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239665 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb533acd-73cb-4f30-a60e-e96debb69f25-agent-certs\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239689 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-tmp-dir\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239719 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d020426a-2902-454c-8b86-7ecc7c666826-iptables-alerter-script\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239783 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d020426a-2902-454c-8b86-7ecc7c666826-host-slash\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239810 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239862 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82eb2931-ff21-4bf4-9589-995055842cb4-serving-cert\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb533acd-73cb-4f30-a60e-e96debb69f25-konnectivity-ca\") pod \"konnectivity-agent-wk8sg\" (UID: \"eb533acd-73cb-4f30-a60e-e96debb69f25\") " pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:34:14.239909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239905 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-log-socket\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239937 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.239969 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240002 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82eb2931-ff21-4bf4-9589-995055842cb4-serving-cert\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240005 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-conf-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240042 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-tmp\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240106 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-tmp\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240130 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-config\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240171 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6dc\" (UniqueName: \"kubernetes.io/projected/d020426a-2902-454c-8b86-7ecc7c666826-kube-api-access-qm6dc\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.240301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240255 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-k8s-cni-cncf-io\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240343 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240351 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-config\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrf85\" (UniqueName: \"kubernetes.io/projected/31e0fa56-e920-4006-91c5-7425cd741ead-kube-api-access-xrf85\") pod \"kas-connection-checker-685895c7c9-lzqpt\" (UID: \"31e0fa56-e920-4006-91c5-7425cd741ead\") " pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240398 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdh9\" (UniqueName: \"kubernetes.io/projected/5af35ed2-0ae8-4679-9c23-419d9a64a2e2-kube-api-access-zcdh9\") pod \"volume-data-source-validator-58c9f5f46d-ctm2h\" (UID: \"5af35ed2-0ae8-4679-9c23-419d9a64a2e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240414 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-stats-auth\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240429 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d0a36a5-f22c-4759-8e09-724db23b7c04-serviceca\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240443 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-sys-fs\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240463 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-ovn\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240502 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240509 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240518 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.240606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240598 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vvn\" (UniqueName: \"kubernetes.io/projected/481f04d7-a2a2-472c-8a0b-744c1ca0e486-kube-api-access-z8vvn\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240632 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-netns\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240652 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240657 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d0a36a5-f22c-4759-8e09-724db23b7c04-serviceca\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240687 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-stats-auth\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240794 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-snapshots\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240801 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-netns\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysconfig\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.240993 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfmr\" (UniqueName: \"kubernetes.io/projected/7cf09ffe-d458-41e7-b9f4-70bf537e64b0-kube-api-access-tmfmr\") pod \"migrator-66df5d496b-jzp7h\" (UID: \"7cf09ffe-d458-41e7-b9f4-70bf537e64b0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cnibin\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.241059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241046 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241068 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-kubelet\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241088 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc6dbf33-7424-4695-9e9a-6a99475477dd-config\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-sys\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-snapshots\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-hostroot\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241160 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtm7\" (UniqueName: \"kubernetes.io/projected/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-kube-api-access-rmtm7\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241183 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-run\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241202 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-var-lib-kubelet\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241217 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241232 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-etc-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241248 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-device-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241265 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241281 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-env-overrides\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241295 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-daemon-config\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.241460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241368 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241480 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241488 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241543 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2h7\" (UniqueName: \"kubernetes.io/projected/99921fd6-6911-4ac5-852d-e90337681ce9-kube-api-access-lc2h7\") pod \"migrator-66df5d496b-p4z7l\" (UID: \"99921fd6-6911-4ac5-852d-e90337681ce9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241562 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-env-overrides\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241570 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-modprobe-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241601 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-daemon-config\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241635 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-kubelet-config\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241641 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241663 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxvm\" (UniqueName: \"kubernetes.io/projected/7b492161-d0aa-42e5-8947-9403654941e3-kube-api-access-rzxvm\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241702 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc6dbf33-7424-4695-9e9a-6a99475477dd-config\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241724 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-socket-dir-parent\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241748 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-kubernetes\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-lib-modules\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241836 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1c67a3a-2cab-4583-976c-a23c07673db0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241882 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.241991 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.242035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1c67a3a-2cab-4583-976c-a23c07673db0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242137 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-service-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242248 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6pr\" (UniqueName: \"kubernetes.io/projected/0d0a36a5-f22c-4759-8e09-724db23b7c04-kube-api-access-8s6pr\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242305 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242355 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-service-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-tmp-dir\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242441 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jzp7h_7cf09ffe-d458-41e7-b9f4-70bf537e64b0/migrator/0.log" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242482 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnd2\" (UniqueName: \"kubernetes.io/projected/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-kube-api-access-rnnd2\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242525 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovn-node-metrics-cert\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242548 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-tmp-dir\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242596 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242634 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78t6\" (UniqueName: \"kubernetes.io/projected/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-kube-api-access-n78t6\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242769 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-crio-socket\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh29q\" (UniqueName: \"kubernetes.io/projected/06076e5c-0467-47fa-a47b-946e170df5bb-kube-api-access-xh29q\") pod \"kas-connection-checker-685895c7c9-xxjw8\" (UID: \"06076e5c-0467-47fa-a47b-946e170df5bb\") " pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242875 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-netd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.242953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242936 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-multus\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.242996 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-host\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243021 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-data-volume\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243132 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff3812dd-9c0e-482d-b760-4540712ff854-cert\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovn-node-metrics-cert\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8kt\" (UniqueName: \"kubernetes.io/projected/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-kube-api-access-rp8kt\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243197 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7kk\" (UniqueName: \"kubernetes.io/projected/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-kube-api-access-zk7kk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243226 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-data-volume\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243288 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-conf\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243332 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243368 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9vg\" (UniqueName: \"kubernetes.io/projected/ff3812dd-9c0e-482d-b760-4540712ff854-kube-api-access-nn9vg\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243403 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff3812dd-9c0e-482d-b760-4540712ff854-cert\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243415 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-bin\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243441 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243511 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-script-lib\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243541 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243600 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243618 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243627 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243662 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-cabundle\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243701 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pztg\" (UniqueName: \"kubernetes.io/projected/0f6b3b2e-5e40-4067-9e29-a891a598286f-kube-api-access-2pztg\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243729 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-kubelet\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243736 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-ovnkube-script-lib\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.243774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243754 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243801 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243818 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243831 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243841 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-systemd\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243884 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243921 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-key\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243949 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-cabundle\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.243959 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244001 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-slash\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244049 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-etc-kubernetes\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-default-certificate\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576ks\" (UniqueName: \"kubernetes.io/projected/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-kube-api-access-576ks\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244106 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244133 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-trusted-ca\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244155 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-system-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244174 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6b3b2e-5e40-4067-9e29-a891a598286f-signing-key\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244174 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzpg\" (UniqueName: \"kubernetes.io/projected/fc6dbf33-7424-4695-9e9a-6a99475477dd-kube-api-access-5jzpg\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244228 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244249 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mv7c\" (UniqueName: \"kubernetes.io/projected/82eb2931-ff21-4bf4-9589-995055842cb4-kube-api-access-5mv7c\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcdz\" (UniqueName: \"kubernetes.io/projected/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b-kube-api-access-mkcdz\") pod \"kas-connection-checker-685895c7c9-nkwv9\" (UID: \"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b\") " pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244314 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4xg\" (UniqueName: \"kubernetes.io/projected/367a2050-9d6e-4486-af6c-1f19921cb8cf-kube-api-access-qk4xg\") pod \"network-check-source-6f78d59f99-bt2cq\" (UID: \"367a2050-9d6e-4486-af6c-1f19921cb8cf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244342 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244353 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-default-certificate\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz8v\" (UniqueName: \"kubernetes.io/projected/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-kube-api-access-4vz8v\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244383 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244415 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b492161-d0aa-42e5-8947-9403654941e3-serving-cert\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244435 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-config\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244453 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-node-log\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dgc\" (UniqueName: \"kubernetes.io/projected/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-kube-api-access-c5dgc\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244500 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-registration-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244537 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244559 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cnibin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-bin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.244962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.244604 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-multus-certs\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245008 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245072 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-config\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245088 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245146 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245181 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-tmp\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245217 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-socket-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245250 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245332 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b492161-d0aa-42e5-8947-9403654941e3-tmp\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b492161-d0aa-42e5-8947-9403654941e3-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245625 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqts\" (UniqueName: \"kubernetes.io/projected/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kube-api-access-mlqts\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245624 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245689 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82eb2931-ff21-4bf4-9589-995055842cb4-trusted-ca\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245724 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-tuned\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b492161-d0aa-42e5-8947-9403654941e3-serving-cert\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245784 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-system-cni-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245800 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-tuned\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245837 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-dbus\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245858 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.246474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.245898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4016e9e3-e3e8-454a-a635-49970015c350-original-pull-secret\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.294813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.294787 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6dc\" (UniqueName: \"kubernetes.io/projected/d020426a-2902-454c-8b86-7ecc7c666826-kube-api-access-qm6dc\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.295096 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.295077 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.295183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.295097 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7rx\" (UniqueName: \"kubernetes.io/projected/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-kube-api-access-fw7rx\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.295261 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.295239 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27524\" (UniqueName: \"kubernetes.io/projected/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-api-access-27524\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.295261 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.295250 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp982\" (UniqueName: \"kubernetes.io/projected/0f0291ae-a1d6-4312-9e07-da45a493466e-kube-api-access-dp982\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.295629 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.295612 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrf85\" (UniqueName: \"kubernetes.io/projected/31e0fa56-e920-4006-91c5-7425cd741ead-kube-api-access-xrf85\") pod \"kas-connection-checker-685895c7c9-lzqpt\" (UID: \"31e0fa56-e920-4006-91c5-7425cd741ead\") " pod="kube-system/kas-connection-checker-685895c7c9-lzqpt" Nov 02 21:34:14.295757 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.295739 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdh9\" (UniqueName: \"kubernetes.io/projected/5af35ed2-0ae8-4679-9c23-419d9a64a2e2-kube-api-access-zcdh9\") pod \"volume-data-source-validator-58c9f5f46d-ctm2h\" (UID: \"5af35ed2-0ae8-4679-9c23-419d9a64a2e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ctm2h" Nov 02 21:34:14.298265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.298248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxvm\" (UniqueName: \"kubernetes.io/projected/7b492161-d0aa-42e5-8947-9403654941e3-kube-api-access-rzxvm\") pod \"insights-operator-7c4986d449-nj4zc\" (UID: \"7b492161-d0aa-42e5-8947-9403654941e3\") " pod="openshift-insights/insights-operator-7c4986d449-nj4zc" Nov 02 21:34:14.299880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.299860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vvn\" (UniqueName: \"kubernetes.io/projected/481f04d7-a2a2-472c-8a0b-744c1ca0e486-kube-api-access-z8vvn\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.299968 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.299914 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.300026 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.299981 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfmr\" (UniqueName: \"kubernetes.io/projected/7cf09ffe-d458-41e7-b9f4-70bf537e64b0-kube-api-access-tmfmr\") pod \"migrator-66df5d496b-jzp7h\" (UID: \"7cf09ffe-d458-41e7-b9f4-70bf537e64b0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jzp7h" Nov 02 21:34:14.300145 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.300125 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8kt\" (UniqueName: \"kubernetes.io/projected/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-kube-api-access-rp8kt\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:14.300547 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.300526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2h7\" (UniqueName: \"kubernetes.io/projected/99921fd6-6911-4ac5-852d-e90337681ce9-kube-api-access-lc2h7\") pod \"migrator-66df5d496b-p4z7l\" (UID: \"99921fd6-6911-4ac5-852d-e90337681ce9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p4z7l" Nov 02 21:34:14.300650 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.300629 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.300903 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.300880 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9vg\" (UniqueName: \"kubernetes.io/projected/ff3812dd-9c0e-482d-b760-4540712ff854-kube-api-access-nn9vg\") pod \"ingress-canary-hsbjx\" (UID: \"ff3812dd-9c0e-482d-b760-4540712ff854\") " pod="openshift-ingress-canary/ingress-canary-hsbjx" Nov 02 21:34:14.300966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.300926 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtm7\" (UniqueName: \"kubernetes.io/projected/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-kube-api-access-rmtm7\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.301279 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pztg\" (UniqueName: \"kubernetes.io/projected/0f6b3b2e-5e40-4067-9e29-a891a598286f-kube-api-access-2pztg\") pod \"service-ca-5fffc97bdf-tzk7v\" (UID: \"0f6b3b2e-5e40-4067-9e29-a891a598286f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tzk7v" Nov 02 21:34:14.301372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301307 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh29q\" (UniqueName: \"kubernetes.io/projected/06076e5c-0467-47fa-a47b-946e170df5bb-kube-api-access-xh29q\") pod \"kas-connection-checker-685895c7c9-xxjw8\" (UID: \"06076e5c-0467-47fa-a47b-946e170df5bb\") " pod="kube-system/kas-connection-checker-685895c7c9-xxjw8" Nov 02 21:34:14.301372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301359 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnd2\" (UniqueName: \"kubernetes.io/projected/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-kube-api-access-rnnd2\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.301523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301497 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7kk\" (UniqueName: \"kubernetes.io/projected/92a05d0e-bf88-42c3-a8d7-f62e1b039a0f-kube-api-access-zk7kk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-cmq47\" (UID: \"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" Nov 02 21:34:14.301642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301623 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.301871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301856 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4xg\" (UniqueName: \"kubernetes.io/projected/367a2050-9d6e-4486-af6c-1f19921cb8cf-kube-api-access-qk4xg\") pod \"network-check-source-6f78d59f99-bt2cq\" (UID: \"367a2050-9d6e-4486-af6c-1f19921cb8cf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bt2cq" Nov 02 21:34:14.301979 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.301961 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzpg\" (UniqueName: \"kubernetes.io/projected/fc6dbf33-7424-4695-9e9a-6a99475477dd-kube-api-access-5jzpg\") pod \"service-ca-operator-544d749c8c-tvss9\" (UID: \"fc6dbf33-7424-4695-9e9a-6a99475477dd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.302450 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576ks\" (UniqueName: \"kubernetes.io/projected/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-kube-api-access-576ks\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78t6\" (UniqueName: \"kubernetes.io/projected/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-kube-api-access-n78t6\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303271 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") pod \"periodic-gathering-w59fz-jvbfg\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303449 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6pr\" (UniqueName: \"kubernetes.io/projected/0d0a36a5-f22c-4759-8e09-724db23b7c04-kube-api-access-8s6pr\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcdz\" (UniqueName: \"kubernetes.io/projected/7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b-kube-api-access-mkcdz\") pod \"kas-connection-checker-685895c7c9-nkwv9\" (UID: \"7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b\") " pod="kube-system/kas-connection-checker-685895c7c9-nkwv9" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303873 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz8v\" (UniqueName: \"kubernetes.io/projected/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-kube-api-access-4vz8v\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303981 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffx9\" (UniqueName: \"kubernetes.io/projected/e987d331-5974-430e-8a6b-d1b1dcbb7fe9-kube-api-access-bffx9\") pod \"network-check-target-9xtbf\" (UID: \"e987d331-5974-430e-8a6b-d1b1dcbb7fe9\") " pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.303982 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mv7c\" (UniqueName: \"kubernetes.io/projected/82eb2931-ff21-4bf4-9589-995055842cb4-kube-api-access-5mv7c\") pod \"console-operator-7cd56b8dbc-kklpk\" (UID: \"82eb2931-ff21-4bf4-9589-995055842cb4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.305351 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dgc\" (UniqueName: \"kubernetes.io/projected/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-kube-api-access-c5dgc\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.306115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.305947 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqts\" (UniqueName: \"kubernetes.io/projected/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kube-api-access-mlqts\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347022 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:14.347120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347049 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.347120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-hosts-file\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.347120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d0a36a5-f22c-4759-8e09-724db23b7c04-host\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.347120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347099 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-systemd-units\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347119 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-systemd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347138 5091 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347155 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/557fdf7f-9d7a-4de7-9819-becbc1fbfb8a-hosts-file\") pod \"node-resolver-4qplz\" (UID: \"557fdf7f-9d7a-4de7-9819-becbc1fbfb8a\") " pod="openshift-dns/node-resolver-4qplz" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d0a36a5-f22c-4759-8e09-724db23b7c04-host\") pod \"node-ca-9njpz\" (UID: \"0d0a36a5-f22c-4759-8e09-724db23b7c04\") " pod="openshift-image-registry/node-ca-9njpz" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347200 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347198 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-systemd-units\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347142 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-var-lib-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347236 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.847214114 +0000 UTC m=+2.009309232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347258 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-systemd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347268 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-var-lib-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347278 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d020426a-2902-454c-8b86-7ecc7c666826-host-slash\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.347312 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347308 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.847279789 +0000 UTC m=+2.009374901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347315 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d020426a-2902-454c-8b86-7ecc7c666826-host-slash\") pod \"iptables-alerter-2nwsl\" (UID: \"d020426a-2902-454c-8b86-7ecc7c666826\") " pod="openshift-network-operator/iptables-alerter-2nwsl" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347334 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-log-socket\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347376 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-conf-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-log-socket\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-k8s-cni-cncf-io\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347417 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347424 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-sys-fs\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347435 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-conf-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347439 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-ovn\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347460 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-ovn\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347461 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347460 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347470 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.847456619 +0000 UTC m=+2.009551722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347487 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347491 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-k8s-cni-cncf-io\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347509 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-netns\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347510 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-sys-fs\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347528 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347538 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-netns\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347556 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-netns\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347573 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysconfig\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cnibin\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347593 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347593 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-run-netns\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347605 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-kubelet\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347618 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.847610783 +0000 UTC m=+2.009705881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347628 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-kubelet\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347648 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-sys\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347654 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysconfig\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347687 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-hostroot\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347694 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-cnibin\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347712 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-run\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-hostroot\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347725 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-sys\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347731 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-var-lib-kubelet\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347754 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-etc-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347761 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-var-lib-kubelet\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347769 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-device-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-run\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347800 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-etc-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347811 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-device-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347826 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347835 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-modprobe-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.347824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347852 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-kubelet-config\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347867 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-socket-dir-parent\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347881 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-kubernetes\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347905 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-lib-modules\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-kubelet-config\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347928 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-multus-socket-dir-parent\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347945 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-modprobe-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347952 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.347951 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-kubernetes\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.347992 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348003 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348005 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348003 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-lib-modules\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348026 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348033 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.848025549 +0000 UTC m=+2.010120647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348062 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.848048554 +0000 UTC m=+2.010143668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348061 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348089 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-crio-socket\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348114 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348128 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-netd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348143 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-multus\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348146 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-run-openvswitch\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-netd\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348171 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-host\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348141 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-crio-socket\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348178 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-multus\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348189 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-conf\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348203 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-bin\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-d\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348231 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-host\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348236 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-kubelet\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348252 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348262 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-kubelet\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348267 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-systemd\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348287 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-slash\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348301 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-etc-kubernetes\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348310 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348318 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-system-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-sysctl-conf\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348355 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.848343615 +0000 UTC m=+2.010438737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348357 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-slash\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348382 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-host-cni-bin\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-etc-kubernetes\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348389 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc532d99-0b1a-40fb-ad7c-42c4cfbb879e-etc-systemd\") pod \"tuned-jxnx6\" (UID: \"fc532d99-0b1a-40fb-ad7c-42c4cfbb879e\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnx6" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348380 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-system-cni-dir\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348425 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348437 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-node-log\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348458 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348477 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c6466fd-6cf7-4d27-bd08-4eaf51c5990e-node-log\") pod \"ovnkube-node-cswj4\" (UID: \"5c6466fd-6cf7-4d27-bd08-4eaf51c5990e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348521 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348547 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-registration-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348571 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.848558278 +0000 UTC m=+2.010653408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : secret "metrics-daemon-secret" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-registration-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348588 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348613 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cnibin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348632 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348634 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-bin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348643 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-cnibin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348658 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-multus-certs\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.348662 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.848654733 +0000 UTC m=+2.010749831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348685 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-var-lib-cni-bin\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348706 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c-host-run-multus-certs\") pod \"multus-cbj8m\" (UID: \"ba1c4210-57de-4ca9-aad7-3fc8a8b8ff0c\") " pod="openshift-multus/multus-cbj8m" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348712 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-socket-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348741 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-socket-dir\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348760 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-system-cni-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348783 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-dbus\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348792 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe41410a-c20e-48c0-b3c3-f531c79c3fe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrs5j\" (UID: \"fe41410a-c20e-48c0-b3c3-f531c79c3fe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrs5j" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348787 5091 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348847 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d0b0db-ceaa-49c5-bff5-16b1d7c370e3-system-cni-dir\") pod \"multus-additional-cni-plugins-w9h49\" (UID: \"56d0b0db-ceaa-49c5-bff5-16b1d7c370e3\") " pod="openshift-multus/multus-additional-cni-plugins-w9h49" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348861 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4016e9e3-e3e8-454a-a635-49970015c350-dbus\") pod \"global-pull-secret-syncer-sr2gb\" (UID: \"4016e9e3-e3e8-454a-a635-49970015c350\") " pod="kube-system/global-pull-secret-syncer-sr2gb" Nov 02 21:34:14.349594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.348953 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-service-ca-bundle\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.351773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.350813 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/481f04d7-a2a2-472c-8a0b-744c1ca0e486-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-29cmv\" (UID: \"481f04d7-a2a2-472c-8a0b-744c1ca0e486\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.379768 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.379744 5091 scope.go:122] "RemoveContainer" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" Nov 02 21:34:14.384447 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.384418 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" Nov 02 21:34:14.442929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.442906 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-p4z7l_99921fd6-6911-4ac5-852d-e90337681ce9/migrator/0.log" Nov 02 21:34:14.643399 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.643368 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-cmq47_92a05d0e-bf88-42c3-a8d7-f62e1b039a0f/kube-storage-version-migrator-operator/0.log" Nov 02 21:34:14.821992 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.821922 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv"] Nov 02 21:34:14.853147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853127 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:14.853250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853164 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.853250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853215 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:14.853349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853257 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:14.853349 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853269 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:14.853349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853283 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:14.853349 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853292 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:14.853349 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853342 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853355 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853346 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853327488 +0000 UTC m=+3.015422608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853390 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853399 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853402 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853426 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853438 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853423671 +0000 UTC m=+3.015518789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853464 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853478 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.8534632 +0000 UTC m=+3.015558321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853481 5091 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853512 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.85349866 +0000 UTC m=+3.015593757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : secret "metrics-daemon-secret" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853525 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853529 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853520048 +0000 UTC m=+3.015615145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853544 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853536735 +0000 UTC m=+3.015631892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853559 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:14.853595 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853564 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853552611 +0000 UTC m=+3.015647716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:14.854088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.853606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:14.854088 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853617 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:14.854088 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853653 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls podName:0f0291ae-a1d6-4312-9e07-da45a493466e nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853640567 +0000 UTC m=+3.015735678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rrvdw" (UID: "0f0291ae-a1d6-4312-9e07-da45a493466e") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:34:14.854088 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853695 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:14.854088 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:14.853724 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.853718029 +0000 UTC m=+3.015813127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:14.890066 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:14.890050 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:34:15.041395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.041367 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/2.log" Nov 02 21:34:15.041860 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.041841 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/1.log" Nov 02 21:34:15.041914 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.041881 5091 generic.go:356] "Generic (PLEG): container finished" podID="82eb2931-ff21-4bf4-9589-995055842cb4" containerID="32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac" exitCode=255 Nov 02 21:34:15.041976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.041955 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" event={"ID":"82eb2931-ff21-4bf4-9589-995055842cb4","Type":"ContainerDied","Data":"32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac"} Nov 02 21:34:15.042021 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.042010 5091 scope.go:122] "RemoveContainer" containerID="18dd01ecbbcab9cdd25b7d43678a471d78257da9d8104919fefa98d14944d860" Nov 02 21:34:15.042377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.042356 5091 scope.go:122] "RemoveContainer" containerID="32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac" Nov 02 21:34:15.042616 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.042589 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:15.043192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.043172 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" event={"ID":"481f04d7-a2a2-472c-8a0b-744c1ca0e486","Type":"ContainerStarted","Data":"504c71b5ed7275c292f596ac597b430ddcd51145be6dafc2dad55b855f639d5f"} Nov 02 21:34:15.634848 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.634824 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:15.801272 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.801239 5091 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:15.862351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862323 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:15.862473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:15.862473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862393 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:15.862473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862449 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:15.862473 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862463 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:15.862473 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862466 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862519 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862504082 +0000 UTC m=+5.024599185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862558 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862473 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862567 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862581 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862578 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862624 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862630 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862625 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862605115 +0000 UTC m=+5.024700230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862696 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.8626636 +0000 UTC m=+5.024758702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862715 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862708761 +0000 UTC m=+5.024803859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862717 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:34:15.862717 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862731 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862725893 +0000 UTC m=+5.024820991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862775 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs podName:b3900b59-124a-4dc5-9cb4-e5e98253eb5a nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862752152 +0000 UTC m=+5.024847269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs") pod "network-metrics-daemon-tfzjr" (UID: "b3900b59-124a-4dc5-9cb4-e5e98253eb5a") : secret "metrics-daemon-secret" not found Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862794 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.862816 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862879 5091 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862882 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862904 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862897289 +0000 UTC m=+5.024992399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:15.863234 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:15.862918 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert podName:c1c67a3a-2cab-4583-976c-a23c07673db0 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:17.862908544 +0000 UTC m=+5.025003664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ptr6r" (UID: "c1c67a3a-2cab-4583-976c-a23c07673db0") : secret "networking-console-plugin-cert" not found Nov 02 21:34:15.864878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.864859 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f0291ae-a1d6-4312-9e07-da45a493466e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rrvdw\" (UID: \"0f0291ae-a1d6-4312-9e07-da45a493466e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:15.880719 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:15.880697 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" Nov 02 21:34:16.048001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:16.047972 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/2.log" Nov 02 21:34:16.048361 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:16.048326 5091 scope.go:122] "RemoveContainer" containerID="32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac" Nov 02 21:34:16.048616 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:16.048596 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:16.266880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:16.266802 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw"] Nov 02 21:34:16.304066 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:16.304016 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0291ae_a1d6_4312_9e07_da45a493466e.slice/crio-364062f3d84435e0aa7c51ed7c343a8e33dddbb8f4d5a6f7f61d3f4830db7adf WatchSource:0}: Error finding container 364062f3d84435e0aa7c51ed7c343a8e33dddbb8f4d5a6f7f61d3f4830db7adf: Status 404 returned error can't find the container with id 364062f3d84435e0aa7c51ed7c343a8e33dddbb8f4d5a6f7f61d3f4830db7adf Nov 02 21:34:17.053655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.053616 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" event={"ID":"0f0291ae-a1d6-4312-9e07-da45a493466e","Type":"ContainerStarted","Data":"364062f3d84435e0aa7c51ed7c343a8e33dddbb8f4d5a6f7f61d3f4830db7adf"} Nov 02 21:34:17.416016 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.415983 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:17.416385 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.416368 5091 scope.go:122] "RemoveContainer" containerID="32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac" Nov 02 21:34:17.416586 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.416565 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:17.579063 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.579030 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:17.884105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884076 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:17.884105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884136 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884154 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884187 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884215 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884232 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884231 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.884260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:17.884321 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884300 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs podName:24c005d3-f7c3-4292-ac2d-ff26ec4065fd nodeName:}" failed. No retries permitted until 2026-11-02 21:34:21.884280256 +0000 UTC m=+9.046375372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs") pod "router-default-cd7d6d4d5-p5mwz" (UID: "24c005d3-f7c3-4292-ac2d-ff26ec4065fd") : secret "router-metrics-certs-default" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884343 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884353 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657747857b-smdws: secret "image-registry-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884375 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884387 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls podName:fc9f5580-bb4a-4c27-8672-949c0507a719 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:21.884377054 +0000 UTC m=+9.046472152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls") pod "image-registry-657747857b-smdws" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719") : secret "image-registry-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884410 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls podName:63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:21.884398497 +0000 UTC m=+9.046493616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls") pod "dns-default-vn9dz" (UID: "63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6") : secret "dns-default-metrics-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884428 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884434 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-554b69489-rlbg8: secret "image-registry-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884453 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls podName:8c425440-6797-483a-b758-10dcc09a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:21.884447205 +0000 UTC m=+9.046542303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls") pod "image-registry-554b69489-rlbg8" (UID: "8c425440-6797-483a-b758-10dcc09a32c7") : secret "image-registry-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884454 5091 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884488 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:21.884476697 +0000 UTC m=+9.046571813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884493 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:17.884743 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:17.884529 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:21.884518897 +0000 UTC m=+9.046613994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:17.886758 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.886734 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3900b59-124a-4dc5-9cb4-e5e98253eb5a-metrics-certs\") pod \"network-metrics-daemon-tfzjr\" (UID: \"b3900b59-124a-4dc5-9cb4-e5e98253eb5a\") " pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:17.886758 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.886739 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1c67a3a-2cab-4583-976c-a23c07673db0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ptr6r\" (UID: \"c1c67a3a-2cab-4583-976c-a23c07673db0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:17.985595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.985524 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" Nov 02 21:34:17.987605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:17.987576 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfzjr" Nov 02 21:34:18.068716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:18.067359 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" event={"ID":"481f04d7-a2a2-472c-8a0b-744c1ca0e486","Type":"ContainerStarted","Data":"2411ff7493cd84c46af7abaf83b6679a519e7c779811dcd636e52d0e16a620f2"} Nov 02 21:34:18.068716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:18.067425 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-29cmv" event={"ID":"481f04d7-a2a2-472c-8a0b-744c1ca0e486","Type":"ContainerStarted","Data":"834ce7d721ba3a641a8c05e7f790c3defd9c9da3b5d0f2ea316257b59deef200"} Nov 02 21:34:18.376938 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:18.376880 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ptr6r"] Nov 02 21:34:18.380055 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:18.380029 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c67a3a_2cab_4583_976c_a23c07673db0.slice/crio-9adccd6f4c8560aabbbc63afa4dba03e42aff47345c8983e183667a346517d42 WatchSource:0}: Error finding container 9adccd6f4c8560aabbbc63afa4dba03e42aff47345c8983e183667a346517d42: Status 404 returned error can't find the container with id 9adccd6f4c8560aabbbc63afa4dba03e42aff47345c8983e183667a346517d42 Nov 02 21:34:18.426549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:18.426472 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tfzjr"] Nov 02 21:34:18.465724 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:18.465693 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3900b59_124a_4dc5_9cb4_e5e98253eb5a.slice/crio-0daf90844e237e6feebf94135f9858761882fa4232edb288e26052a7e077814b WatchSource:0}: Error finding container 0daf90844e237e6feebf94135f9858761882fa4232edb288e26052a7e077814b: Status 404 returned error can't find the container with id 0daf90844e237e6feebf94135f9858761882fa4232edb288e26052a7e077814b Nov 02 21:34:18.847891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:18.847870 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wk8sg" Nov 02 21:34:19.071696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:19.071638 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" event={"ID":"c1c67a3a-2cab-4583-976c-a23c07673db0","Type":"ContainerStarted","Data":"9adccd6f4c8560aabbbc63afa4dba03e42aff47345c8983e183667a346517d42"} Nov 02 21:34:19.072865 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:19.072794 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfzjr" event={"ID":"b3900b59-124a-4dc5-9cb4-e5e98253eb5a","Type":"ContainerStarted","Data":"0daf90844e237e6feebf94135f9858761882fa4232edb288e26052a7e077814b"} Nov 02 21:34:19.074343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:19.074322 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rrvdw" event={"ID":"0f0291ae-a1d6-4312-9e07-da45a493466e","Type":"ContainerStarted","Data":"cda41d10c1105de4e58bf589136fc6ac4b1d04c7d4453b88a829d38c52ebd6f8"} Nov 02 21:34:20.083182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:20.083148 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-ptr6r" event={"ID":"c1c67a3a-2cab-4583-976c-a23c07673db0","Type":"ContainerStarted","Data":"57e5700f04c31e64059ff621305df85e69478efb8f11d8acccd52c4c2b67944e"} Nov 02 21:34:20.085770 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:20.085746 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfzjr" event={"ID":"b3900b59-124a-4dc5-9cb4-e5e98253eb5a","Type":"ContainerStarted","Data":"70aaffbbfa5b6466e791ccb6b7f32c56174f4acd14be140245fc221471f0783e"} Nov 02 21:34:21.090344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.090306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfzjr" event={"ID":"b3900b59-124a-4dc5-9cb4-e5e98253eb5a","Type":"ContainerStarted","Data":"35d71281448009a2a6b33cf6e0de2f7c192ed480bd736d9393695dbb17529921"} Nov 02 21:34:21.923783 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.923743 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:21.923783 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.923795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.923837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.923866 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.923888 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.923934 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:21.923975 5091 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:21.924026 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:34:21.924062 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:21.924049 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:29.924034489 +0000 UTC m=+17.086129613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : configmap "kube-rbac-proxy" not found Nov 02 21:34:21.924416 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:21.924092 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls podName:c5a2b13a-685e-4ba3-8370-e059d7fe5e26 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:29.924072554 +0000 UTC m=+17.086167656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqm" (UID: "c5a2b13a-685e-4ba3-8370-e059d7fe5e26") : secret "insights-runtime-extractor-tls" not found Nov 02 21:34:21.926478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.926453 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"image-registry-657747857b-smdws\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:21.926581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.926453 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"image-registry-554b69489-rlbg8\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:21.926633 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.926611 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6-metrics-tls\") pod \"dns-default-vn9dz\" (UID: \"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6\") " pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:21.926814 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:21.926798 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c005d3-f7c3-4292-ac2d-ff26ec4065fd-metrics-certs\") pod \"router-default-cd7d6d4d5-p5mwz\" (UID: \"24c005d3-f7c3-4292-ac2d-ff26ec4065fd\") " pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:22.183237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.183152 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:22.184721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.184693 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:22.192290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.192267 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:22.193373 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.193350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:22.294324 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.294231 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:22.294504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.294491 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:34:22.659179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.659124 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657747857b-smdws"] Nov 02 21:34:22.660464 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.660375 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554b69489-rlbg8"] Nov 02 21:34:22.660868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.660847 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cd7d6d4d5-p5mwz"] Nov 02 21:34:22.662659 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:22.662633 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9f5580_bb4a_4c27_8672_949c0507a719.slice/crio-50d472e0d6bb9c0e585220067b5ff3aa411d00564438a88f502f8d5a75cfe67b WatchSource:0}: Error finding container 50d472e0d6bb9c0e585220067b5ff3aa411d00564438a88f502f8d5a75cfe67b: Status 404 returned error can't find the container with id 50d472e0d6bb9c0e585220067b5ff3aa411d00564438a88f502f8d5a75cfe67b Nov 02 21:34:22.699201 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:22.699179 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c425440_6797_483a_b758_10dcc09a32c7.slice/crio-88c1134c7d8331b4b3663f561e9f8c1b746a41198a3d9a2111d1728cfa4d4265 WatchSource:0}: Error finding container 88c1134c7d8331b4b3663f561e9f8c1b746a41198a3d9a2111d1728cfa4d4265: Status 404 returned error can't find the container with id 88c1134c7d8331b4b3663f561e9f8c1b746a41198a3d9a2111d1728cfa4d4265 Nov 02 21:34:22.700184 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:22.700160 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c005d3_f7c3_4292_ac2d_ff26ec4065fd.slice/crio-186a58e041f6b2a004fba0294ecb482c292eee3d7277db591f876094f67bf2f8 WatchSource:0}: Error finding container 186a58e041f6b2a004fba0294ecb482c292eee3d7277db591f876094f67bf2f8: Status 404 returned error can't find the container with id 186a58e041f6b2a004fba0294ecb482c292eee3d7277db591f876094f67bf2f8 Nov 02 21:34:22.704939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:22.704921 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vn9dz"] Nov 02 21:34:22.712146 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:22.712121 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ef7a0c_00ee_47c0_9bb4_a8ee237b03b6.slice/crio-a334769255429de143b4df85ffb72abd02017d4a06232f787e5076399bb3ca53 WatchSource:0}: Error finding container a334769255429de143b4df85ffb72abd02017d4a06232f787e5076399bb3ca53: Status 404 returned error can't find the container with id a334769255429de143b4df85ffb72abd02017d4a06232f787e5076399bb3ca53 Nov 02 21:34:23.101108 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.101028 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vn9dz" event={"ID":"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6","Type":"ContainerStarted","Data":"a334769255429de143b4df85ffb72abd02017d4a06232f787e5076399bb3ca53"} Nov 02 21:34:23.102683 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.102638 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" event={"ID":"24c005d3-f7c3-4292-ac2d-ff26ec4065fd","Type":"ContainerStarted","Data":"bd17609a67a561d4296f4620ac260439218ff815cd07712217ba901ab11d3351"} Nov 02 21:34:23.102818 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.102695 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" event={"ID":"24c005d3-f7c3-4292-ac2d-ff26ec4065fd","Type":"ContainerStarted","Data":"186a58e041f6b2a004fba0294ecb482c292eee3d7277db591f876094f67bf2f8"} Nov 02 21:34:23.104406 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.104374 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554b69489-rlbg8" event={"ID":"8c425440-6797-483a-b758-10dcc09a32c7","Type":"ContainerStarted","Data":"89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74"} Nov 02 21:34:23.104524 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.104418 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554b69489-rlbg8" event={"ID":"8c425440-6797-483a-b758-10dcc09a32c7","Type":"ContainerStarted","Data":"88c1134c7d8331b4b3663f561e9f8c1b746a41198a3d9a2111d1728cfa4d4265"} Nov 02 21:34:23.106043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.106003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657747857b-smdws" event={"ID":"fc9f5580-bb4a-4c27-8672-949c0507a719","Type":"ContainerStarted","Data":"056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f"} Nov 02 21:34:23.106043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.106043 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657747857b-smdws" event={"ID":"fc9f5580-bb4a-4c27-8672-949c0507a719","Type":"ContainerStarted","Data":"50d472e0d6bb9c0e585220067b5ff3aa411d00564438a88f502f8d5a75cfe67b"} Nov 02 21:34:23.185548 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.185523 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:23.187938 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.187906 5091 patch_prober.go:29] interesting pod/router-default-cd7d6d4d5-p5mwz container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 02 21:34:23.187938 ip-10-0-128-36 kubenswrapper[5091]: [-]has-synced failed: reason withheld Nov 02 21:34:23.187938 ip-10-0-128-36 kubenswrapper[5091]: [+]process-running ok Nov 02 21:34:23.187938 ip-10-0-128-36 kubenswrapper[5091]: healthz check failed Nov 02 21:34:23.188127 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.187960 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" podUID="24c005d3-f7c3-4292-ac2d-ff26ec4065fd" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 02 21:34:23.506336 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:23.506304 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:24.110477 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:24.110443 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:34:24.186645 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:24.186615 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:25.115454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:25.115373 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vn9dz" event={"ID":"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6","Type":"ContainerStarted","Data":"686da5f7ac49458ef89d10a4a5fb517fb78e2917fda733538c7e4c716299e61a"} Nov 02 21:34:25.115454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:25.115425 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vn9dz" event={"ID":"63ef7a0c-00ee-47c0-9bb4-a8ee237b03b6","Type":"ContainerStarted","Data":"8ede34633bb2c5c5adb01cec65da53ba84e5bee5ec8f46ec1d8da3813c589ef7"} Nov 02 21:34:26.119435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:26.119404 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:34:26.119896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:26.119704 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:27.969771 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:27.969744 5091 scope.go:122] "RemoveContainer" containerID="32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac" Nov 02 21:34:28.129173 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:28.129147 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:34:28.129566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:28.129551 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/2.log" Nov 02 21:34:28.129689 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:28.129582 5091 generic.go:356] "Generic (PLEG): container finished" podID="82eb2931-ff21-4bf4-9589-995055842cb4" containerID="9d54f479af276c6d97bc433a1b57fda2995b7d811420d89e0b76bfc7d608b73f" exitCode=255 Nov 02 21:34:28.129737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:28.129662 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" event={"ID":"82eb2931-ff21-4bf4-9589-995055842cb4","Type":"ContainerDied","Data":"9d54f479af276c6d97bc433a1b57fda2995b7d811420d89e0b76bfc7d608b73f"} Nov 02 21:34:28.129771 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:28.129739 5091 scope.go:122] "RemoveContainer" containerID="32b4c92c1d412da663158a99e1de99e8a4800c61bfbe3cdd1ae6ba8d0caa08ac" Nov 02 21:34:28.130057 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:28.130036 5091 scope.go:122] "RemoveContainer" containerID="9d54f479af276c6d97bc433a1b57fda2995b7d811420d89e0b76bfc7d608b73f" Nov 02 21:34:28.130282 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:28.130263 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:29.098272 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:29.098229 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:29.135731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:29.135705 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:34:29.992942 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:29.992909 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:29.993088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:29.992966 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:29.993456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:29.993435 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:29.995241 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:29.995217 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5a2b13a-685e-4ba3-8370-e059d7fe5e26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqm\" (UID: \"c5a2b13a-685e-4ba3-8370-e059d7fe5e26\") " pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:30.079368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:30.079338 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:30.079501 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:30.079474 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:34:30.080585 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:30.080560 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-cd7d6d4d5-p5mwz" Nov 02 21:34:30.295083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:30.295012 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5tqm" Nov 02 21:34:30.592532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:30.592458 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657747857b-smdws"] Nov 02 21:34:30.758698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:30.758638 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b5tqm"] Nov 02 21:34:30.762817 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:30.762781 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a2b13a_685e_4ba3_8370_e059d7fe5e26.slice/crio-5e70a04bdd9e7fac8388159c21ea6620e9b6ae813a79a611165fdfc6e138c1a8 WatchSource:0}: Error finding container 5e70a04bdd9e7fac8388159c21ea6620e9b6ae813a79a611165fdfc6e138c1a8: Status 404 returned error can't find the container with id 5e70a04bdd9e7fac8388159c21ea6620e9b6ae813a79a611165fdfc6e138c1a8 Nov 02 21:34:31.124748 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:31.124720 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cswj4" Nov 02 21:34:31.144735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:31.144707 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqm" event={"ID":"c5a2b13a-685e-4ba3-8370-e059d7fe5e26","Type":"ContainerStarted","Data":"bfbff50ba07f00cb0d79f42d9b7139230926eaa3edfde8060e12090646d2c4d0"} Nov 02 21:34:31.144873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:31.144759 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqm" event={"ID":"c5a2b13a-685e-4ba3-8370-e059d7fe5e26","Type":"ContainerStarted","Data":"5e70a04bdd9e7fac8388159c21ea6620e9b6ae813a79a611165fdfc6e138c1a8"} Nov 02 21:34:32.150644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:32.150611 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqm" event={"ID":"c5a2b13a-685e-4ba3-8370-e059d7fe5e26","Type":"ContainerStarted","Data":"c46032e39de9025f0309e10ad34478940391389dda0386a29cc18aa274d8fb4d"} Nov 02 21:34:34.159543 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:34.159511 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqm" event={"ID":"c5a2b13a-685e-4ba3-8370-e059d7fe5e26","Type":"ContainerStarted","Data":"f3655c178841bf83d173fd2f42c8076b74fb0a50dfe085898658846316c27017"} Nov 02 21:34:35.802101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:35.802061 5091 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:35.802487 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:35.802417 5091 scope.go:122] "RemoveContainer" containerID="9d54f479af276c6d97bc433a1b57fda2995b7d811420d89e0b76bfc7d608b73f" Nov 02 21:34:35.802620 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:35.802600 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:36.126267 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:36.126239 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vn9dz" Nov 02 21:34:37.111092 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:37.111053 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:34:37.416513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:37.416471 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:37.416884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:37.416869 5091 scope.go:122] "RemoveContainer" containerID="9d54f479af276c6d97bc433a1b57fda2995b7d811420d89e0b76bfc7d608b73f" Nov 02 21:34:37.417066 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:37.417048 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kklpk_openshift-console-operator(82eb2931-ff21-4bf4-9589-995055842cb4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" Nov 02 21:34:39.942687 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:39.942636 5091 patch_prober.go:29] interesting pod/image-registry-554b69489-rlbg8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 21:34:39.943116 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:39.942712 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-554b69489-rlbg8" podUID="8c425440-6797-483a-b758-10dcc09a32c7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:34:40.457916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.457878 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bhzhl"] Nov 02 21:34:40.489011 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.488988 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.578937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.578901 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-root\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.578944 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.578972 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-sys\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.579109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-tls\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.579151 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-textfile\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.579179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8be30588-9291-4133-82a4-3a6df47e2b4a-metrics-client-ca\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579306 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.579277 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-wtmp\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579350 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.579304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgwd\" (UniqueName: \"kubernetes.io/projected/8be30588-9291-4133-82a4-3a6df47e2b4a-kube-api-access-gbgwd\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.579350 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.579339 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.598327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.598300 5091 patch_prober.go:29] interesting pod/image-registry-657747857b-smdws container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 21:34:40.598470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.598356 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-657747857b-smdws" podUID="fc9f5580-bb4a-4c27-8672-949c0507a719" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:34:40.651461 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.651432 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:34:40.651461 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.651448 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xsz5x\"" Nov 02 21:34:40.651461 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.651448 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:34:40.651691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.651530 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:34:40.651691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.651605 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:34:40.680775 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680743 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-tls\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.680887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680782 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-textfile\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.680887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8be30588-9291-4133-82a4-3a6df47e2b4a-metrics-client-ca\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.680887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-wtmp\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.680887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgwd\" (UniqueName: \"kubernetes.io/projected/8be30588-9291-4133-82a4-3a6df47e2b4a-kube-api-access-gbgwd\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680905 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680949 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-root\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.680976 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681001 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-sys\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681051 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-root\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-wtmp\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681081 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be30588-9291-4133-82a4-3a6df47e2b4a-sys\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681155 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-textfile\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681509 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681472 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8be30588-9291-4133-82a4-3a6df47e2b4a-metrics-client-ca\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.681619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.681531 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.683107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.683088 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-tls\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.683210 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.683191 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8be30588-9291-4133-82a4-3a6df47e2b4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.737206 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.737148 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgwd\" (UniqueName: \"kubernetes.io/projected/8be30588-9291-4133-82a4-3a6df47e2b4a-kube-api-access-gbgwd\") pod \"node-exporter-bhzhl\" (UID: \"8be30588-9291-4133-82a4-3a6df47e2b4a\") " pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.844378 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:40.844348 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhzhl" Nov 02 21:34:40.881800 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:34:40.881770 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be30588_9291_4133_82a4_3a6df47e2b4a.slice/crio-8d1cf77f25b06cb35fbfd1e804d9bf22a92302ce7214349c4178b89e845f8b7a WatchSource:0}: Error finding container 8d1cf77f25b06cb35fbfd1e804d9bf22a92302ce7214349c4178b89e845f8b7a: Status 404 returned error can't find the container with id 8d1cf77f25b06cb35fbfd1e804d9bf22a92302ce7214349c4178b89e845f8b7a Nov 02 21:34:41.187420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:41.187383 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhzhl" event={"ID":"8be30588-9291-4133-82a4-3a6df47e2b4a","Type":"ContainerStarted","Data":"8d1cf77f25b06cb35fbfd1e804d9bf22a92302ce7214349c4178b89e845f8b7a"} Nov 02 21:34:43.197166 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:43.197135 5091 generic.go:356] "Generic (PLEG): container finished" podID="8be30588-9291-4133-82a4-3a6df47e2b4a" containerID="6edaef06d9497d04aba0447f578666b8e0865372575341791fc38d267f8e3c34" exitCode=0 Nov 02 21:34:43.197601 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:43.197211 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhzhl" event={"ID":"8be30588-9291-4133-82a4-3a6df47e2b4a","Type":"ContainerDied","Data":"6edaef06d9497d04aba0447f578666b8e0865372575341791fc38d267f8e3c34"} Nov 02 21:34:44.203207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:44.203171 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhzhl" event={"ID":"8be30588-9291-4133-82a4-3a6df47e2b4a","Type":"ContainerStarted","Data":"6657a85c417e501e6a53d88e053e0de05b3c1da40da67630b8f99d4145d8668b"} Nov 02 21:34:44.203579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:44.203251 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhzhl" event={"ID":"8be30588-9291-4133-82a4-3a6df47e2b4a","Type":"ContainerStarted","Data":"f7dba3e369115760215c70d0d5f0fe83172cb3fe30921b7f0848abcddf880baa"} Nov 02 21:34:44.368802 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:44.368733 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bhzhl" podStartSLOduration=2.405364151 podStartE2EDuration="4.368716287s" totalImagesPullingTime="1.884501244s" totalInitContainerRuntime="78.850892ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:44.368268232 +0000 UTC m=+31.530363353" watchObservedRunningTime="2026-11-02 21:34:44.368716287 +0000 UTC m=+31.530811408" Nov 02 21:34:45.120688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:45.120643 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:34:47.582156 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:47.582128 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9xtbf" Nov 02 21:34:50.597586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:50.597558 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:53.969945 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:53.969878 5091 scope.go:122] "RemoveContainer" containerID="9d54f479af276c6d97bc433a1b57fda2995b7d811420d89e0b76bfc7d608b73f" Nov 02 21:34:54.248275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:54.248208 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:34:54.248275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:54.248262 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" event={"ID":"82eb2931-ff21-4bf4-9589-995055842cb4","Type":"ContainerStarted","Data":"c96b2b437579e8082c1666fd9bcc408dcb401652dcef47c5b47180abf0dc6f32"} Nov 02 21:34:54.248538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:54.248519 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:54.249883 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:54.249858 5091 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-kklpk container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Nov 02 21:34:54.250009 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:54.249902 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" podUID="82eb2931-ff21-4bf4-9589-995055842cb4" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Nov 02 21:34:54.459780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:54.459747 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-554b69489-rlbg8"] Nov 02 21:34:55.256513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.256469 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kklpk" Nov 02 21:34:55.615426 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.615323 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-657747857b-smdws" podUID="fc9f5580-bb4a-4c27-8672-949c0507a719" containerName="registry" containerID="cri-o://056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f" gracePeriod=30 Nov 02 21:34:55.866033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.865972 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:55.910327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910298 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910377 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910414 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910635 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910478 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910635 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910513 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910635 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910552 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:55.910964 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.910913 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:55.913070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.913046 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "kube-api-access-ghzsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:55.913192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.913122 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:55.913192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.913156 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:55.913192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.913154 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:55.913349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.913198 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:55.917559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:55.917434 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:34:56.011045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011018 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") pod \"fc9f5580-bb4a-4c27-8672-949c0507a719\" (UID: \"fc9f5580-bb4a-4c27-8672-949c0507a719\") " Nov 02 21:34:56.011241 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011224 5091 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc9f5580-bb4a-4c27-8672-949c0507a719-ca-trust-extracted\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011247 5091 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-tls\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011260 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-trusted-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011276 5091 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-image-registry-private-configuration\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011291 5091 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-bound-sa-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011305 5091 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc9f5580-bb4a-4c27-8672-949c0507a719-installation-pull-secrets\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011471 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011319 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzsm\" (UniqueName: \"kubernetes.io/projected/fc9f5580-bb4a-4c27-8672-949c0507a719-kube-api-access-ghzsm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.011471 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.011357 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates" pod "fc9f5580-bb4a-4c27-8672-949c0507a719" (UID: "fc9f5580-bb4a-4c27-8672-949c0507a719"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:56.112058 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.112026 5091 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc9f5580-bb4a-4c27-8672-949c0507a719-registry-certificates\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:34:56.257742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.257712 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc9f5580-bb4a-4c27-8672-949c0507a719" containerID="056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f" exitCode=0 Nov 02 21:34:56.258214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.257756 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657747857b-smdws" Nov 02 21:34:56.258214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.257784 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657747857b-smdws" event={"ID":"fc9f5580-bb4a-4c27-8672-949c0507a719","Type":"ContainerDied","Data":"056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f"} Nov 02 21:34:56.258214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.257831 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657747857b-smdws" event={"ID":"fc9f5580-bb4a-4c27-8672-949c0507a719","Type":"ContainerDied","Data":"50d472e0d6bb9c0e585220067b5ff3aa411d00564438a88f502f8d5a75cfe67b"} Nov 02 21:34:56.258214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.257851 5091 scope.go:122] "RemoveContainer" containerID="056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f" Nov 02 21:34:56.271980 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.271957 5091 scope.go:122] "RemoveContainer" containerID="056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f" Nov 02 21:34:56.272266 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:34:56.272245 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f\": container with ID starting with 056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f not found: ID does not exist" containerID="056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f" Nov 02 21:34:56.272306 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.272275 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f"} err="failed to get container status \"056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f\": rpc error: code = NotFound desc = could not find container \"056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f\": container with ID starting with 056ceb965f47f7b3ce3a3c1c7ec2cdfdb317a12ceaf385db4f1bb17abd11d10f not found: ID does not exist" Nov 02 21:34:56.423342 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.423308 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657747857b-smdws"] Nov 02 21:34:56.473027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.472989 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-657747857b-smdws"] Nov 02 21:34:56.974826 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:34:56.974795 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc9f5580-bb4a-4c27-8672-949c0507a719" path="/var/lib/kubelet/pods/fc9f5580-bb4a-4c27-8672-949c0507a719/volumes" Nov 02 21:35:07.110876 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:35:07.110835 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:35:17.340125 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:17.340099 5091 generic.go:356] "Generic (PLEG): container finished" podID="7b492161-d0aa-42e5-8947-9403654941e3" containerID="0ab1721b66640b5863d1ee90c7ac4a27f74cc9708f814f27edd94fcf6608536c" exitCode=0 Nov 02 21:35:17.340456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:17.340171 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" event={"ID":"7b492161-d0aa-42e5-8947-9403654941e3","Type":"ContainerDied","Data":"0ab1721b66640b5863d1ee90c7ac4a27f74cc9708f814f27edd94fcf6608536c"} Nov 02 21:35:17.340515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:17.340463 5091 scope.go:122] "RemoveContainer" containerID="0ab1721b66640b5863d1ee90c7ac4a27f74cc9708f814f27edd94fcf6608536c" Nov 02 21:35:18.346190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.346110 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nj4zc" event={"ID":"7b492161-d0aa-42e5-8947-9403654941e3","Type":"ContainerStarted","Data":"b39cd724bbf7c2039d8bf475e7101695b2d60532cbb1d9cb18294f6671c727dc"} Nov 02 21:35:18.347609 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.347586 5091 generic.go:356] "Generic (PLEG): container finished" podID="92a05d0e-bf88-42c3-a8d7-f62e1b039a0f" containerID="6ff3fbec60d82dd7d51a8279e0d8a12af077968904d93ab5593f18a804966577" exitCode=0 Nov 02 21:35:18.347711 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.347626 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" event={"ID":"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f","Type":"ContainerDied","Data":"6ff3fbec60d82dd7d51a8279e0d8a12af077968904d93ab5593f18a804966577"} Nov 02 21:35:18.347876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.347864 5091 scope.go:122] "RemoveContainer" containerID="6ff3fbec60d82dd7d51a8279e0d8a12af077968904d93ab5593f18a804966577" Nov 02 21:35:18.449030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.448994 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-w59fz-jvbfg"] Nov 02 21:35:18.449338 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.449233 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" podUID="f1404835-f6fb-400f-b107-476157b43c7d" containerName="insights-gathering" containerID="cri-o://39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb" gracePeriod=30 Nov 02 21:35:18.694258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.694233 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:35:18.766821 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.766790 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") pod \"f1404835-f6fb-400f-b107-476157b43c7d\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " Nov 02 21:35:18.766983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.766840 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") pod \"f1404835-f6fb-400f-b107-476157b43c7d\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " Nov 02 21:35:18.766983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.766864 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") pod \"f1404835-f6fb-400f-b107-476157b43c7d\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " Nov 02 21:35:18.766983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.766891 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") pod \"f1404835-f6fb-400f-b107-476157b43c7d\" (UID: \"f1404835-f6fb-400f-b107-476157b43c7d\") " Nov 02 21:35:18.767192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.767157 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path" pod "f1404835-f6fb-400f-b107-476157b43c7d" (UID: "f1404835-f6fb-400f-b107-476157b43c7d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:18.767309 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.767288 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle" pod "f1404835-f6fb-400f-b107-476157b43c7d" (UID: "f1404835-f6fb-400f-b107-476157b43c7d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:18.767353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.767297 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle" pod "f1404835-f6fb-400f-b107-476157b43c7d" (UID: "f1404835-f6fb-400f-b107-476157b43c7d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:18.769143 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.769117 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p" pod "f1404835-f6fb-400f-b107-476157b43c7d" (UID: "f1404835-f6fb-400f-b107-476157b43c7d"). InnerVolumeSpecName "kube-api-access-fkl8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:18.867454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.867392 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkl8p\" (UniqueName: \"kubernetes.io/projected/f1404835-f6fb-400f-b107-476157b43c7d-kube-api-access-fkl8p\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:18.867454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.867420 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f1404835-f6fb-400f-b107-476157b43c7d-archives-path\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:18.867454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.867431 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-trusted-ca-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:18.867454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:18.867440 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1404835-f6fb-400f-b107-476157b43c7d-service-ca-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.352754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.352713 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-cmq47" event={"ID":"92a05d0e-bf88-42c3-a8d7-f62e1b039a0f","Type":"ContainerStarted","Data":"0173294b0e7327cd0229f1b07afceb9b7bfd39cada1b619b5183d7b4a7de48e7"} Nov 02 21:35:19.354006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.353981 5091 generic.go:356] "Generic (PLEG): container finished" podID="f1404835-f6fb-400f-b107-476157b43c7d" containerID="39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb" exitCode=2 Nov 02 21:35:19.354124 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.354019 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" Nov 02 21:35:19.354124 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.354021 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" event={"ID":"f1404835-f6fb-400f-b107-476157b43c7d","Type":"ContainerDied","Data":"39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb"} Nov 02 21:35:19.354207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.354157 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w59fz-jvbfg" event={"ID":"f1404835-f6fb-400f-b107-476157b43c7d","Type":"ContainerDied","Data":"31d8e9ff0ab355ce7bcd4277d664e9208682381039ef9656cc3414ac8ab62134"} Nov 02 21:35:19.354207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.354182 5091 scope.go:122] "RemoveContainer" containerID="39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb" Nov 02 21:35:19.362003 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.361987 5091 scope.go:122] "RemoveContainer" containerID="39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb" Nov 02 21:35:19.362255 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:35:19.362235 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb\": container with ID starting with 39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb not found: ID does not exist" containerID="39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb" Nov 02 21:35:19.362317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.362263 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb"} err="failed to get container status \"39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb\": rpc error: code = NotFound desc = could not find container \"39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb\": container with ID starting with 39933c568c5484bca001b6d0e636032b3fb95615a10eedb94bf87973e7927cdb not found: ID does not exist" Nov 02 21:35:19.479310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.479267 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-554b69489-rlbg8" podUID="8c425440-6797-483a-b758-10dcc09a32c7" containerName="registry" containerID="cri-o://89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74" gracePeriod=30 Nov 02 21:35:19.575050 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.575018 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-w59fz-jvbfg"] Nov 02 21:35:19.578688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.578652 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-w59fz-jvbfg"] Nov 02 21:35:19.718117 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.718088 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:35:19.775737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775705 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.775866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775755 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.775866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775794 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.775866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775844 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.776004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775885 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.776004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775918 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.776004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775951 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.776004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.775999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") pod \"8c425440-6797-483a-b758-10dcc09a32c7\" (UID: \"8c425440-6797-483a-b758-10dcc09a32c7\") " Nov 02 21:35:19.776515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.776446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:19.776515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.776496 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:19.778302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.778275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "kube-api-access-22hsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:19.778601 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.778580 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:19.778720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.778694 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:19.778780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.778711 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:19.778780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.778731 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:19.784828 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.784798 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted" pod "8c425440-6797-483a-b758-10dcc09a32c7" (UID: "8c425440-6797-483a-b758-10dcc09a32c7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:19.876846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876818 5091 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-image-registry-private-configuration\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.876846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876841 5091 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c425440-6797-483a-b758-10dcc09a32c7-installation-pull-secrets\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.876846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876853 5091 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-registry-tls\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.877047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876863 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22hsk\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-kube-api-access-22hsk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.877047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876871 5091 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-registry-certificates\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.877047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876880 5091 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c425440-6797-483a-b758-10dcc09a32c7-ca-trust-extracted\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.877047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876889 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c425440-6797-483a-b758-10dcc09a32c7-trusted-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.877047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:19.876897 5091 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c425440-6797-483a-b758-10dcc09a32c7-bound-sa-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.358973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.358941 5091 generic.go:356] "Generic (PLEG): container finished" podID="8c425440-6797-483a-b758-10dcc09a32c7" containerID="89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74" exitCode=0 Nov 02 21:35:20.359460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.358991 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554b69489-rlbg8" Nov 02 21:35:20.359460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.359022 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554b69489-rlbg8" event={"ID":"8c425440-6797-483a-b758-10dcc09a32c7","Type":"ContainerDied","Data":"89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74"} Nov 02 21:35:20.359460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.359072 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554b69489-rlbg8" event={"ID":"8c425440-6797-483a-b758-10dcc09a32c7","Type":"ContainerDied","Data":"88c1134c7d8331b4b3663f561e9f8c1b746a41198a3d9a2111d1728cfa4d4265"} Nov 02 21:35:20.359460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.359094 5091 scope.go:122] "RemoveContainer" containerID="89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74" Nov 02 21:35:20.367763 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.367741 5091 scope.go:122] "RemoveContainer" containerID="89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74" Nov 02 21:35:20.368017 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:35:20.367998 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74\": container with ID starting with 89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74 not found: ID does not exist" containerID="89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74" Nov 02 21:35:20.368088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.368029 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74"} err="failed to get container status \"89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74\": rpc error: code = NotFound desc = could not find container \"89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74\": container with ID starting with 89c547effc9c2d232cf1d800c57e98febf3546c500e315683a4740aa530fbe74 not found: ID does not exist" Nov 02 21:35:20.568719 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.568687 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-554b69489-rlbg8"] Nov 02 21:35:20.618801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.618775 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-554b69489-rlbg8"] Nov 02 21:35:20.974804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.974735 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c425440-6797-483a-b758-10dcc09a32c7" path="/var/lib/kubelet/pods/8c425440-6797-483a-b758-10dcc09a32c7/volumes" Nov 02 21:35:20.975104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:20.975092 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1404835-f6fb-400f-b107-476157b43c7d" path="/var/lib/kubelet/pods/f1404835-f6fb-400f-b107-476157b43c7d/volumes" Nov 02 21:35:28.396948 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:28.396916 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc6dbf33-7424-4695-9e9a-6a99475477dd" containerID="41de14bd08fc21d26c7cad0531ce14ff089c7be54de8a94e71dcb3c64d9bce07" exitCode=0 Nov 02 21:35:28.397322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:28.396987 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" event={"ID":"fc6dbf33-7424-4695-9e9a-6a99475477dd","Type":"ContainerDied","Data":"41de14bd08fc21d26c7cad0531ce14ff089c7be54de8a94e71dcb3c64d9bce07"} Nov 02 21:35:28.397322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:28.397271 5091 scope.go:122] "RemoveContainer" containerID="41de14bd08fc21d26c7cad0531ce14ff089c7be54de8a94e71dcb3c64d9bce07" Nov 02 21:35:29.402774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:35:29.402737 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tvss9" event={"ID":"fc6dbf33-7424-4695-9e9a-6a99475477dd","Type":"ContainerStarted","Data":"c0b4633dff834d0186748a66ca6554528f29f390268c3fac1cbb2533b9b54ed0"} Nov 02 21:35:37.110307 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:35:37.110272 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:36:07.110827 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:36:07.110790 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:36:37.111189 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:36:37.111110 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:37:07.111731 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:37:07.111689 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:37:37.111490 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:37:37.111448 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:38:07.111510 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:38:07.111472 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:38:25.008303 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008269 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp"] Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008834 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8c425440-6797-483a-b758-10dcc09a32c7" containerName="registry" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008859 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c425440-6797-483a-b758-10dcc09a32c7" containerName="registry" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008884 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f1404835-f6fb-400f-b107-476157b43c7d" containerName="insights-gathering" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008892 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1404835-f6fb-400f-b107-476157b43c7d" containerName="insights-gathering" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008908 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fc9f5580-bb4a-4c27-8672-949c0507a719" containerName="registry" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008915 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9f5580-bb4a-4c27-8672-949c0507a719" containerName="registry" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.008993 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1404835-f6fb-400f-b107-476157b43c7d" containerName="insights-gathering" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.009008 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9f5580-bb4a-4c27-8672-949c0507a719" containerName="registry" Nov 02 21:38:25.009044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.009023 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c425440-6797-483a-b758-10dcc09a32c7" containerName="registry" Nov 02 21:38:25.012233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.012210 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.064179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.064154 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 21:38:25.096644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.096617 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.096760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.096650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.096760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.096688 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk44g\" (UniqueName: \"kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.113760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.113735 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 21:38:25.113907 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.113890 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-w2rv8\"" Nov 02 21:38:25.114804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.114787 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp"] Nov 02 21:38:25.198102 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.198076 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.198219 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.198110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk44g\" (UniqueName: \"kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.198219 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.198198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.198456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.198435 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.198540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.198506 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.254354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.254321 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk44g\" (UniqueName: \"kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.328523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.328465 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:25.694660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:25.694630 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp"] Nov 02 21:38:25.697593 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:25.697560 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6496d765_6951_4e7c_9a24_cd5330faa085.slice/crio-e0ee86e5e839933c320a921fc49515ef2125b3677c9270b1010e2855efe0887e WatchSource:0}: Error finding container e0ee86e5e839933c320a921fc49515ef2125b3677c9270b1010e2855efe0887e: Status 404 returned error can't find the container with id e0ee86e5e839933c320a921fc49515ef2125b3677c9270b1010e2855efe0887e Nov 02 21:38:26.025119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:26.025030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" event={"ID":"6496d765-6951-4e7c-9a24-cd5330faa085","Type":"ContainerStarted","Data":"e0ee86e5e839933c320a921fc49515ef2125b3677c9270b1010e2855efe0887e"} Nov 02 21:38:26.940942 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:26.940906 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9"] Nov 02 21:38:26.944709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:26.944684 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.010104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.009897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.010104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.010019 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjn2g\" (UniqueName: \"kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.010104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.010057 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.046117 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.046086 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-zkqvp\"" Nov 02 21:38:27.047275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.047251 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9"] Nov 02 21:38:27.048159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.048133 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p"] Nov 02 21:38:27.051828 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.051809 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.104292 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.104248 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-g4wbl\"" Nov 02 21:38:27.105180 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.105151 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p"] Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.110660 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnwx\" (UniqueName: \"kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.110753 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.110808 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.110857 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.110892 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.110923 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjn2g\" (UniqueName: \"kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.111274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.111173 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.111653 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.111454 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.138201 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.138173 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs"] Nov 02 21:38:27.141631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.141613 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.167030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.167001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjn2g\" (UniqueName: \"kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.198054 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.197943 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs"] Nov 02 21:38:27.203985 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.203961 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-bfv7k\"" Nov 02 21:38:27.212042 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.212164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnwx\" (UniqueName: \"kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.212164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmmb\" (UniqueName: \"kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.212290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212162 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.212290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.212290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212212 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.212468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212387 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.212529 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.212487 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.262650 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.262625 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:27.268802 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.268782 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnwx\" (UniqueName: \"kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.313513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.313478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmmb\" (UniqueName: \"kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.313699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.313553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.313699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.313593 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.314060 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.314032 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.314164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.314103 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.370428 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.370395 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmmb\" (UniqueName: \"kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.370592 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.370540 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:27.460035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.459964 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:27.639893 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.639869 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9"] Nov 02 21:38:27.642933 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:27.642902 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29375575_7546_48fe_8f56_b7d54b94f3ff.slice/crio-0098ab3486c6aba5b5b6be8995a9ea4b557dbfea7c27ea0c94634bd7f9c9811f WatchSource:0}: Error finding container 0098ab3486c6aba5b5b6be8995a9ea4b557dbfea7c27ea0c94634bd7f9c9811f: Status 404 returned error can't find the container with id 0098ab3486c6aba5b5b6be8995a9ea4b557dbfea7c27ea0c94634bd7f9c9811f Nov 02 21:38:27.750517 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.750448 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p"] Nov 02 21:38:27.753264 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:27.753237 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e8cdb3_af05_422e_b817_dde0d47a6cca.slice/crio-1e71e4a59be7fff2b34cd7dff23708658b0b43c51caf32cea0679f9b81f6a0fe WatchSource:0}: Error finding container 1e71e4a59be7fff2b34cd7dff23708658b0b43c51caf32cea0679f9b81f6a0fe: Status 404 returned error can't find the container with id 1e71e4a59be7fff2b34cd7dff23708658b0b43c51caf32cea0679f9b81f6a0fe Nov 02 21:38:27.831108 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:27.830983 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs"] Nov 02 21:38:27.833977 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:27.833949 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f12ac1a_fa6c_4f88_93ba_3a49d76825fd.slice/crio-a9d04e7884fc1acfd9baf6a4307ea73f19f9217e83c2957f9b43196ab95b6ea5 WatchSource:0}: Error finding container a9d04e7884fc1acfd9baf6a4307ea73f19f9217e83c2957f9b43196ab95b6ea5: Status 404 returned error can't find the container with id a9d04e7884fc1acfd9baf6a4307ea73f19f9217e83c2957f9b43196ab95b6ea5 Nov 02 21:38:28.034728 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.034624 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" event={"ID":"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd","Type":"ContainerStarted","Data":"a9d04e7884fc1acfd9baf6a4307ea73f19f9217e83c2957f9b43196ab95b6ea5"} Nov 02 21:38:28.036050 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.036015 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" event={"ID":"30e8cdb3-af05-422e-b817-dde0d47a6cca","Type":"ContainerStarted","Data":"1e71e4a59be7fff2b34cd7dff23708658b0b43c51caf32cea0679f9b81f6a0fe"} Nov 02 21:38:28.037249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.037219 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" event={"ID":"29375575-7546-48fe-8f56-b7d54b94f3ff","Type":"ContainerStarted","Data":"0098ab3486c6aba5b5b6be8995a9ea4b557dbfea7c27ea0c94634bd7f9c9811f"} Nov 02 21:38:28.866651 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.866615 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58"] Nov 02 21:38:28.873774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.871357 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:28.928728 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.928695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:28.929041 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.928966 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftjr\" (UniqueName: \"kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:28.929152 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.929107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:28.973259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.973229 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-52j4l\"" Nov 02 21:38:28.977014 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:28.976989 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58"] Nov 02 21:38:29.025965 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.025931 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q"] Nov 02 21:38:29.033960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.033938 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.035143 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.034456 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.035249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.035194 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.035320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.035274 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftjr\" (UniqueName: \"kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.035984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.034829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.035984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.035908 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.049474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.049444 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q"] Nov 02 21:38:29.090111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.090085 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-njfrv\"" Nov 02 21:38:29.094868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.094815 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftjr\" (UniqueName: \"kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.136582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.136326 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.136582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.136373 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.136582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.136401 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhtf\" (UniqueName: \"kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.193469 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.193437 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:29.239059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.237965 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.239059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.238007 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.239059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.238035 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhtf\" (UniqueName: \"kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.239059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.238732 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.239059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.239011 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.307544 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.307501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhtf\" (UniqueName: \"kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.357800 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.357766 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:29.589724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.589689 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58"] Nov 02 21:38:29.594970 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:29.594936 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f91f085_eebb_4431_9254_3b06b6da5b8e.slice/crio-29ebf3f66bbe215037d8c2306e787b946412bc7e2d399f2901c4d639dada074a WatchSource:0}: Error finding container 29ebf3f66bbe215037d8c2306e787b946412bc7e2d399f2901c4d639dada074a: Status 404 returned error can't find the container with id 29ebf3f66bbe215037d8c2306e787b946412bc7e2d399f2901c4d639dada074a Nov 02 21:38:29.743785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:29.743657 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q"] Nov 02 21:38:29.749633 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:29.749576 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1255dd_ae74_4293_90ef_d5ee26f2ddc7.slice/crio-a0c4970b2b3da5a4d83543caa52c93d57b94ceb6a77676f6dc3ff868689deed7 WatchSource:0}: Error finding container a0c4970b2b3da5a4d83543caa52c93d57b94ceb6a77676f6dc3ff868689deed7: Status 404 returned error can't find the container with id a0c4970b2b3da5a4d83543caa52c93d57b94ceb6a77676f6dc3ff868689deed7 Nov 02 21:38:30.050959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:30.050874 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" event={"ID":"9f91f085-eebb-4431-9254-3b06b6da5b8e","Type":"ContainerStarted","Data":"29ebf3f66bbe215037d8c2306e787b946412bc7e2d399f2901c4d639dada074a"} Nov 02 21:38:30.053032 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:30.053000 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" event={"ID":"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7","Type":"ContainerStarted","Data":"a0c4970b2b3da5a4d83543caa52c93d57b94ceb6a77676f6dc3ff868689deed7"} Nov 02 21:38:30.982638 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:30.982607 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx"] Nov 02 21:38:30.986414 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:30.986391 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.039130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.039097 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qzh4z\"" Nov 02 21:38:31.056570 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056538 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056603 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056626 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056648 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056685 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056701 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hfw5\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056724 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.057015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.056761 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.064183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.064099 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 21:38:31.093566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.093488 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx"] Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157525 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157592 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157638 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157664 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157710 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157744 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157783 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157808 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hfw5\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.157841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.158461 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.158703 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.159322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.158948 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.160793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.160752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.160905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.160834 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.163649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.163628 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.166743 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.166715 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.223779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.219309 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hfw5\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.226545 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.226498 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5wrqx\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.303777 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.303146 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:31.769899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:31.769813 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx"] Nov 02 21:38:31.773990 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:31.773957 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d69c63c_9787_40b3_a692_6f723a96cefa.slice/crio-fea88ce646abbfd9a9b2143b42a6e59f1572469494c18362f26623b359fcc845 WatchSource:0}: Error finding container fea88ce646abbfd9a9b2143b42a6e59f1572469494c18362f26623b359fcc845: Status 404 returned error can't find the container with id fea88ce646abbfd9a9b2143b42a6e59f1572469494c18362f26623b359fcc845 Nov 02 21:38:32.070399 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:32.070273 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" event={"ID":"6d69c63c-9787-40b3-a692-6f723a96cefa","Type":"ContainerStarted","Data":"fea88ce646abbfd9a9b2143b42a6e59f1572469494c18362f26623b359fcc845"} Nov 02 21:38:37.115519 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:37.115464 5091 generic.go:356] "Generic (PLEG): container finished" podID="6496d765-6951-4e7c-9a24-cd5330faa085" containerID="b6289c51da72c7772e0bee4d31b787280f7b47c6023798be0ff706695463f7dd" exitCode=0 Nov 02 21:38:37.116053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:37.115555 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" event={"ID":"6496d765-6951-4e7c-9a24-cd5330faa085","Type":"ContainerDied","Data":"b6289c51da72c7772e0bee4d31b787280f7b47c6023798be0ff706695463f7dd"} Nov 02 21:38:37.145505 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:38:37.145469 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:38:38.128398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:38.128351 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" event={"ID":"9f91f085-eebb-4431-9254-3b06b6da5b8e","Type":"ContainerStarted","Data":"12d566854686f2036dce80082daa80946a6f1366beda84f020a63c5fbd51fe88"} Nov 02 21:38:38.905140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:38.905097 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:38:38.905255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:38.905199 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:38:38.905299 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:38.905258 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:38:39.137725 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.137664 5091 generic.go:356] "Generic (PLEG): container finished" podID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerID="12d566854686f2036dce80082daa80946a6f1366beda84f020a63c5fbd51fe88" exitCode=0 Nov 02 21:38:39.138175 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.137794 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" event={"ID":"9f91f085-eebb-4431-9254-3b06b6da5b8e","Type":"ContainerDied","Data":"12d566854686f2036dce80082daa80946a6f1366beda84f020a63c5fbd51fe88"} Nov 02 21:38:39.141693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.141647 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerID="785bfe41bdc9126350951d5654a0e4c314528e202c9e93fbc0a226c7f3c27c9c" exitCode=0 Nov 02 21:38:39.141807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.141716 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" event={"ID":"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7","Type":"ContainerDied","Data":"785bfe41bdc9126350951d5654a0e4c314528e202c9e93fbc0a226c7f3c27c9c"} Nov 02 21:38:39.143760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.143736 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" event={"ID":"6d69c63c-9787-40b3-a692-6f723a96cefa","Type":"ContainerStarted","Data":"ec78dd3a1e8ccad850d45e5ddab65490c1547cbe9d3d9d0d8dfec3ede1e2abb4"} Nov 02 21:38:39.304487 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.304403 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:39.310460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.310429 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:39.601160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:39.601054 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podStartSLOduration=2.476993446 podStartE2EDuration="9.601035519s" totalImagesPullingTime="7.124042073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:39.485186829 +0000 UTC m=+266.647281960" watchObservedRunningTime="2026-11-02 21:38:39.601035519 +0000 UTC m=+266.763130640" Nov 02 21:38:40.149932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.149898 5091 generic.go:356] "Generic (PLEG): container finished" podID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerID="9ce09b07a43d7a1600a2e72eca4997eff9a653cd1dd8bf5521017d0269436f51" exitCode=0 Nov 02 21:38:40.150367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.149975 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" event={"ID":"30e8cdb3-af05-422e-b817-dde0d47a6cca","Type":"ContainerDied","Data":"9ce09b07a43d7a1600a2e72eca4997eff9a653cd1dd8bf5521017d0269436f51"} Nov 02 21:38:40.152453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.152432 5091 generic.go:356] "Generic (PLEG): container finished" podID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerID="8223e0bf4030c023bdb4a302b81fe89a0d174de86cad480557196c2a974f8cc2" exitCode=0 Nov 02 21:38:40.152539 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.152479 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" event={"ID":"29375575-7546-48fe-8f56-b7d54b94f3ff","Type":"ContainerDied","Data":"8223e0bf4030c023bdb4a302b81fe89a0d174de86cad480557196c2a974f8cc2"} Nov 02 21:38:40.154437 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.154324 5091 generic.go:356] "Generic (PLEG): container finished" podID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerID="8c0af42facd1033bfeb83f4fb17d7f28627b17849da018648987778eb5991402" exitCode=0 Nov 02 21:38:40.154437 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.154409 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" event={"ID":"9f91f085-eebb-4431-9254-3b06b6da5b8e","Type":"ContainerDied","Data":"8c0af42facd1033bfeb83f4fb17d7f28627b17849da018648987778eb5991402"} Nov 02 21:38:40.156125 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.156104 5091 generic.go:356] "Generic (PLEG): container finished" podID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerID="e609c0486f8c7a38aa73a8a0af4aa4831f670b9c977d8d49578c8d80d8c9a912" exitCode=0 Nov 02 21:38:40.156203 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.156185 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" event={"ID":"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd","Type":"ContainerDied","Data":"e609c0486f8c7a38aa73a8a0af4aa4831f670b9c977d8d49578c8d80d8c9a912"} Nov 02 21:38:40.156384 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.156364 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:40.157268 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:40.157247 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:38:41.163093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:41.163062 5091 generic.go:356] "Generic (PLEG): container finished" podID="6496d765-6951-4e7c-9a24-cd5330faa085" containerID="f96677d0b74a099f370f3d27fe4421873fc71b6650465c481b1b94f974dc09be" exitCode=0 Nov 02 21:38:41.163503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:41.163149 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" event={"ID":"6496d765-6951-4e7c-9a24-cd5330faa085","Type":"ContainerDied","Data":"f96677d0b74a099f370f3d27fe4421873fc71b6650465c481b1b94f974dc09be"} Nov 02 21:38:43.177793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.177747 5091 generic.go:356] "Generic (PLEG): container finished" podID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerID="e88cdfb83a22d158d88ced1c5fd8131fed7ec5d83bf7c9a95318b31e11ea8bf8" exitCode=0 Nov 02 21:38:43.178471 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.178439 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" event={"ID":"30e8cdb3-af05-422e-b817-dde0d47a6cca","Type":"ContainerDied","Data":"e88cdfb83a22d158d88ced1c5fd8131fed7ec5d83bf7c9a95318b31e11ea8bf8"} Nov 02 21:38:43.181130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.181063 5091 generic.go:356] "Generic (PLEG): container finished" podID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerID="8224cf3050b90193f4fe0484e40df7d7d783309e6f88ef4bad81515da64b57c7" exitCode=0 Nov 02 21:38:43.181388 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.181357 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" event={"ID":"29375575-7546-48fe-8f56-b7d54b94f3ff","Type":"ContainerDied","Data":"8224cf3050b90193f4fe0484e40df7d7d783309e6f88ef4bad81515da64b57c7"} Nov 02 21:38:43.183428 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.183403 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerID="477b68bae9b9263ba0a213ad9cfe5c326c9a50ceceff2517e81f698dd5d8e4d2" exitCode=0 Nov 02 21:38:43.183527 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.183485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" event={"ID":"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7","Type":"ContainerDied","Data":"477b68bae9b9263ba0a213ad9cfe5c326c9a50ceceff2517e81f698dd5d8e4d2"} Nov 02 21:38:43.186372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.186347 5091 generic.go:356] "Generic (PLEG): container finished" podID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerID="34057290193facd7ef21fd7053963d57059a74cd4ced6bc9ae4625442e95d4e5" exitCode=0 Nov 02 21:38:43.186452 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:43.186405 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" event={"ID":"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd","Type":"ContainerDied","Data":"34057290193facd7ef21fd7053963d57059a74cd4ced6bc9ae4625442e95d4e5"} Nov 02 21:38:52.260646 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:52.260612 5091 generic.go:356] "Generic (PLEG): container finished" podID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerID="37f06b3b49e2d1a0354693254820c932137f562a97fc958b32bcbb13766bfa18" exitCode=0 Nov 02 21:38:52.261636 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:52.261554 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" event={"ID":"9f91f085-eebb-4431-9254-3b06b6da5b8e","Type":"ContainerDied","Data":"37f06b3b49e2d1a0354693254820c932137f562a97fc958b32bcbb13766bfa18"} Nov 02 21:38:53.269221 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.269185 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" event={"ID":"30e8cdb3-af05-422e-b817-dde0d47a6cca","Type":"ContainerStarted","Data":"c43f153f104c0e20b72fed7ae6c33c193b71f68388d42844781566b29ebf6d06"} Nov 02 21:38:53.272134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.272104 5091 generic.go:356] "Generic (PLEG): container finished" podID="6496d765-6951-4e7c-9a24-cd5330faa085" containerID="8427b5f98dd1f45235666cb312ebbf1bdab4b3050e7fc57a008b0a5ed27fae91" exitCode=0 Nov 02 21:38:53.272488 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.272466 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" event={"ID":"6496d765-6951-4e7c-9a24-cd5330faa085","Type":"ContainerDied","Data":"8427b5f98dd1f45235666cb312ebbf1bdab4b3050e7fc57a008b0a5ed27fae91"} Nov 02 21:38:53.385511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.385385 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" podStartSLOduration=1.7830314459999999 podStartE2EDuration="26.385367866s" totalImagesPullingTime="24.60233642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:53.383281499 +0000 UTC m=+280.545376620" watchObservedRunningTime="2026-11-02 21:38:53.385367866 +0000 UTC m=+280.547462977" Nov 02 21:38:53.465087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.463186 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:53.581551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.581513 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle\") pod \"9f91f085-eebb-4431-9254-3b06b6da5b8e\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " Nov 02 21:38:53.581734 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.581580 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr\" (UniqueName: \"kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr\") pod \"9f91f085-eebb-4431-9254-3b06b6da5b8e\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " Nov 02 21:38:53.581734 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.581651 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util\") pod \"9f91f085-eebb-4431-9254-3b06b6da5b8e\" (UID: \"9f91f085-eebb-4431-9254-3b06b6da5b8e\") " Nov 02 21:38:53.582691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.582632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle" pod "9f91f085-eebb-4431-9254-3b06b6da5b8e" (UID: "9f91f085-eebb-4431-9254-3b06b6da5b8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:53.584966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.584904 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr" pod "9f91f085-eebb-4431-9254-3b06b6da5b8e" (UID: "9f91f085-eebb-4431-9254-3b06b6da5b8e"). InnerVolumeSpecName "kube-api-access-7ftjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:53.587736 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.587712 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util" pod "9f91f085-eebb-4431-9254-3b06b6da5b8e" (UID: "9f91f085-eebb-4431-9254-3b06b6da5b8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:53.683568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.683252 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:53.683568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.683445 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftjr\" (UniqueName: \"kubernetes.io/projected/9f91f085-eebb-4431-9254-3b06b6da5b8e-kube-api-access-7ftjr\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:53.683568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:53.683465 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f91f085-eebb-4431-9254-3b06b6da5b8e-util\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:54.280708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.280647 5091 generic.go:356] "Generic (PLEG): container finished" podID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerID="c43f153f104c0e20b72fed7ae6c33c193b71f68388d42844781566b29ebf6d06" exitCode=0 Nov 02 21:38:54.281158 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.280791 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" event={"ID":"30e8cdb3-af05-422e-b817-dde0d47a6cca","Type":"ContainerDied","Data":"c43f153f104c0e20b72fed7ae6c33c193b71f68388d42844781566b29ebf6d06"} Nov 02 21:38:54.283764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.283728 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" event={"ID":"9f91f085-eebb-4431-9254-3b06b6da5b8e","Type":"ContainerDied","Data":"29ebf3f66bbe215037d8c2306e787b946412bc7e2d399f2901c4d639dada074a"} Nov 02 21:38:54.283899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.283782 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ebf3f66bbe215037d8c2306e787b946412bc7e2d399f2901c4d639dada074a" Nov 02 21:38:54.283899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.283850 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718xd58" Nov 02 21:38:54.449474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.449449 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:54.596372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.596292 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle\") pod \"6496d765-6951-4e7c-9a24-cd5330faa085\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " Nov 02 21:38:54.596372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.596340 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g\" (UniqueName: \"kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g\") pod \"6496d765-6951-4e7c-9a24-cd5330faa085\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " Nov 02 21:38:54.596593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.596411 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util\") pod \"6496d765-6951-4e7c-9a24-cd5330faa085\" (UID: \"6496d765-6951-4e7c-9a24-cd5330faa085\") " Nov 02 21:38:54.597188 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.597132 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle" pod "6496d765-6951-4e7c-9a24-cd5330faa085" (UID: "6496d765-6951-4e7c-9a24-cd5330faa085"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:54.599796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.599750 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g" pod "6496d765-6951-4e7c-9a24-cd5330faa085" (UID: "6496d765-6951-4e7c-9a24-cd5330faa085"). InnerVolumeSpecName "kube-api-access-sk44g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:54.602268 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.602241 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util" pod "6496d765-6951-4e7c-9a24-cd5330faa085" (UID: "6496d765-6951-4e7c-9a24-cd5330faa085"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:54.697549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.697514 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-util\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:54.697549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.697548 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6496d765-6951-4e7c-9a24-cd5330faa085-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:54.697812 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:54.697564 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk44g\" (UniqueName: \"kubernetes.io/projected/6496d765-6951-4e7c-9a24-cd5330faa085-kube-api-access-sk44g\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:55.290617 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.290590 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" Nov 02 21:38:55.291150 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.291129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd6snp" event={"ID":"6496d765-6951-4e7c-9a24-cd5330faa085","Type":"ContainerDied","Data":"e0ee86e5e839933c320a921fc49515ef2125b3677c9270b1010e2855efe0887e"} Nov 02 21:38:55.291228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.291170 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ee86e5e839933c320a921fc49515ef2125b3677c9270b1010e2855efe0887e" Nov 02 21:38:55.415071 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.414854 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:55.605468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.605391 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx\" (UniqueName: \"kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx\") pod \"30e8cdb3-af05-422e-b817-dde0d47a6cca\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " Nov 02 21:38:55.605591 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.605482 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util\") pod \"30e8cdb3-af05-422e-b817-dde0d47a6cca\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " Nov 02 21:38:55.605591 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.605543 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle\") pod \"30e8cdb3-af05-422e-b817-dde0d47a6cca\" (UID: \"30e8cdb3-af05-422e-b817-dde0d47a6cca\") " Nov 02 21:38:55.606058 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.606030 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle" pod "30e8cdb3-af05-422e-b817-dde0d47a6cca" (UID: "30e8cdb3-af05-422e-b817-dde0d47a6cca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:55.607474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.607449 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx" pod "30e8cdb3-af05-422e-b817-dde0d47a6cca" (UID: "30e8cdb3-af05-422e-b817-dde0d47a6cca"). InnerVolumeSpecName "kube-api-access-5dnwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:55.610735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.610715 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util" pod "30e8cdb3-af05-422e-b817-dde0d47a6cca" (UID: "30e8cdb3-af05-422e-b817-dde0d47a6cca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:55.707042 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.707012 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:55.707173 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.707047 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnwx\" (UniqueName: \"kubernetes.io/projected/30e8cdb3-af05-422e-b817-dde0d47a6cca-kube-api-access-5dnwx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:55.707173 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:55.707062 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e8cdb3-af05-422e-b817-dde0d47a6cca-util\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:56.296463 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.296378 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerID="ac7beeffa07b3bb4a8601647f70c4aceea1d572796d867aab0c9c1651adf072b" exitCode=0 Nov 02 21:38:56.296849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.296457 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" event={"ID":"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7","Type":"ContainerDied","Data":"ac7beeffa07b3bb4a8601647f70c4aceea1d572796d867aab0c9c1651adf072b"} Nov 02 21:38:56.298481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.298452 5091 generic.go:356] "Generic (PLEG): container finished" podID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerID="e2381838916e9c120cba40cece9d302b43dc6d26f27942b5380c1239a3f3bb18" exitCode=0 Nov 02 21:38:56.298608 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.298542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" event={"ID":"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd","Type":"ContainerDied","Data":"e2381838916e9c120cba40cece9d302b43dc6d26f27942b5380c1239a3f3bb18"} Nov 02 21:38:56.300361 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.300342 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" Nov 02 21:38:56.300468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.300366 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccqs6p" event={"ID":"30e8cdb3-af05-422e-b817-dde0d47a6cca","Type":"ContainerDied","Data":"1e71e4a59be7fff2b34cd7dff23708658b0b43c51caf32cea0679f9b81f6a0fe"} Nov 02 21:38:56.300468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.300403 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e71e4a59be7fff2b34cd7dff23708658b0b43c51caf32cea0679f9b81f6a0fe" Nov 02 21:38:56.302347 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.302324 5091 generic.go:356] "Generic (PLEG): container finished" podID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerID="1a646111d0d90efaa13c380cb4f38f9c904e8850fd6fcfb25934e50b59905aff" exitCode=0 Nov 02 21:38:56.302454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:56.302353 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" event={"ID":"29375575-7546-48fe-8f56-b7d54b94f3ff","Type":"ContainerDied","Data":"1a646111d0d90efaa13c380cb4f38f9c904e8850fd6fcfb25934e50b59905aff"} Nov 02 21:38:57.456375 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.456341 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:57.494611 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.494590 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:57.497278 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.497260 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:57.523160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523143 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb\" (UniqueName: \"kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb\") pod \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " Nov 02 21:38:57.523248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523172 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util\") pod \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " Nov 02 21:38:57.523248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523195 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g\" (UniqueName: \"kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g\") pod \"29375575-7546-48fe-8f56-b7d54b94f3ff\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " Nov 02 21:38:57.523248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523233 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle\") pod \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " Nov 02 21:38:57.523412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523268 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util\") pod \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " Nov 02 21:38:57.523412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle\") pod \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\" (UID: \"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd\") " Nov 02 21:38:57.523412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523372 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf\" (UniqueName: \"kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf\") pod \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\" (UID: \"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7\") " Nov 02 21:38:57.523412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523405 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util\") pod \"29375575-7546-48fe-8f56-b7d54b94f3ff\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " Nov 02 21:38:57.523607 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle\") pod \"29375575-7546-48fe-8f56-b7d54b94f3ff\" (UID: \"29375575-7546-48fe-8f56-b7d54b94f3ff\") " Nov 02 21:38:57.523703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.523655 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle" pod "ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" (UID: "ec1255dd-ae74-4293-90ef-d5ee26f2ddc7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:57.524309 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.524280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle" pod "4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" (UID: "4f12ac1a-fa6c-4f88-93ba-3a49d76825fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:57.524386 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.524364 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle" pod "29375575-7546-48fe-8f56-b7d54b94f3ff" (UID: "29375575-7546-48fe-8f56-b7d54b94f3ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:57.525590 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.525564 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g" pod "29375575-7546-48fe-8f56-b7d54b94f3ff" (UID: "29375575-7546-48fe-8f56-b7d54b94f3ff"). InnerVolumeSpecName "kube-api-access-jjn2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:57.525721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.525687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb" pod "4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" (UID: "4f12ac1a-fa6c-4f88-93ba-3a49d76825fd"). InnerVolumeSpecName "kube-api-access-tgmmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:57.526848 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.526819 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf" pod "ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" (UID: "ec1255dd-ae74-4293-90ef-d5ee26f2ddc7"). InnerVolumeSpecName "kube-api-access-vnhtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:57.529710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.529690 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util" pod "4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" (UID: "4f12ac1a-fa6c-4f88-93ba-3a49d76825fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:57.530640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.530622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util" pod "ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" (UID: "ec1255dd-ae74-4293-90ef-d5ee26f2ddc7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:57.532274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.532249 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util" pod "29375575-7546-48fe-8f56-b7d54b94f3ff" (UID: "29375575-7546-48fe-8f56-b7d54b94f3ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:57.624245 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624211 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhtf\" (UniqueName: \"kubernetes.io/projected/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-kube-api-access-vnhtf\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624245 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624244 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-util\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624258 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29375575-7546-48fe-8f56-b7d54b94f3ff-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624270 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgmmb\" (UniqueName: \"kubernetes.io/projected/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-kube-api-access-tgmmb\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624281 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-util\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624293 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjn2g\" (UniqueName: \"kubernetes.io/projected/29375575-7546-48fe-8f56-b7d54b94f3ff-kube-api-access-jjn2g\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624304 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624315 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1255dd-ae74-4293-90ef-d5ee26f2ddc7-util\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:57.624457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:57.624326 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f12ac1a-fa6c-4f88-93ba-3a49d76825fd-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:58.314226 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.314192 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" Nov 02 21:38:58.314226 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.314206 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajgvk9" event={"ID":"29375575-7546-48fe-8f56-b7d54b94f3ff","Type":"ContainerDied","Data":"0098ab3486c6aba5b5b6be8995a9ea4b557dbfea7c27ea0c94634bd7f9c9811f"} Nov 02 21:38:58.314499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.314260 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0098ab3486c6aba5b5b6be8995a9ea4b557dbfea7c27ea0c94634bd7f9c9811f" Nov 02 21:38:58.316202 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.316172 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" Nov 02 21:38:58.316326 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.316167 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cr52q" event={"ID":"ec1255dd-ae74-4293-90ef-d5ee26f2ddc7","Type":"ContainerDied","Data":"a0c4970b2b3da5a4d83543caa52c93d57b94ceb6a77676f6dc3ff868689deed7"} Nov 02 21:38:58.316326 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.316294 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c4970b2b3da5a4d83543caa52c93d57b94ceb6a77676f6dc3ff868689deed7" Nov 02 21:38:58.318191 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.318164 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" event={"ID":"4f12ac1a-fa6c-4f88-93ba-3a49d76825fd","Type":"ContainerDied","Data":"a9d04e7884fc1acfd9baf6a4307ea73f19f9217e83c2957f9b43196ab95b6ea5"} Nov 02 21:38:58.318282 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.318200 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9d04e7884fc1acfd9baf6a4307ea73f19f9217e83c2957f9b43196ab95b6ea5" Nov 02 21:38:58.318282 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:58.318248 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vcqfs" Nov 02 21:38:59.239915 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.239878 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb"] Nov 02 21:38:59.240403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240308 5091 cpu_manager.go:414] "containerMap: removing container" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="extract" Nov 02 21:38:59.240403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240327 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="extract" Nov 02 21:38:59.240403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240368 5091 cpu_manager.go:414] "containerMap: removing container" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="util" Nov 02 21:38:59.240403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240376 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="util" Nov 02 21:38:59.240403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240389 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="pull" Nov 02 21:38:59.240403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240396 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240411 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240419 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240435 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240442 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240452 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240459 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240468 5091 cpu_manager.go:414] "containerMap: removing container" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240475 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240488 5091 cpu_manager.go:414] "containerMap: removing container" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240495 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240508 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240515 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240524 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240531 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240545 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240552 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240561 5091 cpu_manager.go:414] "containerMap: removing container" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240568 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240577 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240584 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="util" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240595 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240602 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240612 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240618 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240650 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240659 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240686 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240695 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="extract" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240708 5091 cpu_manager.go:414] "containerMap: removing container" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="pull" Nov 02 21:38:59.240740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240715 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="pull" Nov 02 21:38:59.242029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240806 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e8cdb3-af05-422e-b817-dde0d47a6cca" containerName="extract" Nov 02 21:38:59.242029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240822 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="29375575-7546-48fe-8f56-b7d54b94f3ff" containerName="extract" Nov 02 21:38:59.242029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240834 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6496d765-6951-4e7c-9a24-cd5330faa085" containerName="extract" Nov 02 21:38:59.242029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240842 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1255dd-ae74-4293-90ef-d5ee26f2ddc7" containerName="extract" Nov 02 21:38:59.242029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240851 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f12ac1a-fa6c-4f88-93ba-3a49d76825fd" containerName="extract" Nov 02 21:38:59.242029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.240860 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f91f085-eebb-4431-9254-3b06b6da5b8e" containerName="extract" Nov 02 21:38:59.243748 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.243728 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.295935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.295910 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb"] Nov 02 21:38:59.335824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.335800 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wxs\" (UniqueName: \"kubernetes.io/projected/05845a1f-26ab-4af4-b0cf-02fcf83aa677-kube-api-access-g4wxs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb\" (UID: \"05845a1f-26ab-4af4-b0cf-02fcf83aa677\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.335954 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.335840 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05845a1f-26ab-4af4-b0cf-02fcf83aa677-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb\" (UID: \"05845a1f-26ab-4af4-b0cf-02fcf83aa677\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.343701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.343665 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:38:59.343768 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.343714 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 21:38:59.343814 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.343797 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8srjv\"" Nov 02 21:38:59.436476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.436445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wxs\" (UniqueName: \"kubernetes.io/projected/05845a1f-26ab-4af4-b0cf-02fcf83aa677-kube-api-access-g4wxs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb\" (UID: \"05845a1f-26ab-4af4-b0cf-02fcf83aa677\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.436579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.436488 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05845a1f-26ab-4af4-b0cf-02fcf83aa677-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb\" (UID: \"05845a1f-26ab-4af4-b0cf-02fcf83aa677\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.436864 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.436849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05845a1f-26ab-4af4-b0cf-02fcf83aa677-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb\" (UID: \"05845a1f-26ab-4af4-b0cf-02fcf83aa677\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.492484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.492422 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wxs\" (UniqueName: \"kubernetes.io/projected/05845a1f-26ab-4af4-b0cf-02fcf83aa677-kube-api-access-g4wxs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb\" (UID: \"05845a1f-26ab-4af4-b0cf-02fcf83aa677\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.553407 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.553373 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" Nov 02 21:38:59.973660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:38:59.973637 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb"] Nov 02 21:38:59.976057 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:38:59.976033 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05845a1f_26ab_4af4_b0cf_02fcf83aa677.slice/crio-ce0a056fccab7c2c3822ccff6c9f5c190fc84b9fce59ae1e16d76d033681b2ca WatchSource:0}: Error finding container ce0a056fccab7c2c3822ccff6c9f5c190fc84b9fce59ae1e16d76d033681b2ca: Status 404 returned error can't find the container with id ce0a056fccab7c2c3822ccff6c9f5c190fc84b9fce59ae1e16d76d033681b2ca Nov 02 21:39:00.327823 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:00.327742 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" event={"ID":"05845a1f-26ab-4af4-b0cf-02fcf83aa677","Type":"ContainerStarted","Data":"ce0a056fccab7c2c3822ccff6c9f5c190fc84b9fce59ae1e16d76d033681b2ca"} Nov 02 21:39:07.111324 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:39:07.111273 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:39:08.759181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.759150 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv"] Nov 02 21:39:08.765389 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.762637 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.808606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.808576 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fdcfe05-bce2-46dd-9152-d8e08c669411-manifests\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.808761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.808609 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.808761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.808630 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.808761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.808647 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fdcfe05-bce2-46dd-9152-d8e08c669411-tmp\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.808878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.808802 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.808878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.808851 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp65l\" (UniqueName: \"kubernetes.io/projected/7fdcfe05-bce2-46dd-9152-d8e08c669411-kube-api-access-pp65l\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.821716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.821696 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 21:39:08.865153 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.865130 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 21:39:08.865262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.865189 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 21:39:08.865262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.865205 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 21:39:08.865262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.865236 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cf4br\"" Nov 02 21:39:08.865262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.865246 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 21:39:08.876702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.876679 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv"] Nov 02 21:39:08.909299 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp65l\" (UniqueName: \"kubernetes.io/projected/7fdcfe05-bce2-46dd-9152-d8e08c669411-kube-api-access-pp65l\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.909394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909306 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fdcfe05-bce2-46dd-9152-d8e08c669411-manifests\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.909394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.909394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909358 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.909394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909383 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fdcfe05-bce2-46dd-9152-d8e08c669411-tmp\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.909608 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909429 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.909759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.909740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fdcfe05-bce2-46dd-9152-d8e08c669411-manifests\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.911941 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.911915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.912022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.911936 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.912022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.911981 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fdcfe05-bce2-46dd-9152-d8e08c669411-tmp\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.912217 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.912193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fdcfe05-bce2-46dd-9152-d8e08c669411-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:08.966424 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:08.966396 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp65l\" (UniqueName: \"kubernetes.io/projected/7fdcfe05-bce2-46dd-9152-d8e08c669411-kube-api-access-pp65l\") pod \"opendatahub-operator-controller-manager-67f87dc46d-9xbzv\" (UID: \"7fdcfe05-bce2-46dd-9152-d8e08c669411\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:09.077976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:09.077904 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:09.562192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:09.562166 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv"] Nov 02 21:39:09.563574 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:09.563551 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdcfe05_bce2_46dd_9152_d8e08c669411.slice/crio-91b77c99aeeac4b5b83f2abb902f979cce7287871609caff2fdfe9178b65e8eb WatchSource:0}: Error finding container 91b77c99aeeac4b5b83f2abb902f979cce7287871609caff2fdfe9178b65e8eb: Status 404 returned error can't find the container with id 91b77c99aeeac4b5b83f2abb902f979cce7287871609caff2fdfe9178b65e8eb Nov 02 21:39:10.372250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:10.372215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" event={"ID":"7fdcfe05-bce2-46dd-9152-d8e08c669411","Type":"ContainerStarted","Data":"91b77c99aeeac4b5b83f2abb902f979cce7287871609caff2fdfe9178b65e8eb"} Nov 02 21:39:13.006993 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:13.006970 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:39:13.007346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:13.007242 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:39:13.389250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:13.389213 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" event={"ID":"7fdcfe05-bce2-46dd-9152-d8e08c669411","Type":"ContainerStarted","Data":"9fd635fb6dbbae1adb9b84c26d4904e2acafab9d6d328c460de7e21925c20d13"} Nov 02 21:39:14.394790 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:14.394752 5091 generic.go:356] "Generic (PLEG): container finished" podID="7fdcfe05-bce2-46dd-9152-d8e08c669411" containerID="9fd635fb6dbbae1adb9b84c26d4904e2acafab9d6d328c460de7e21925c20d13" exitCode=0 Nov 02 21:39:14.395164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:14.394843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" event={"ID":"7fdcfe05-bce2-46dd-9152-d8e08c669411","Type":"ContainerDied","Data":"9fd635fb6dbbae1adb9b84c26d4904e2acafab9d6d328c460de7e21925c20d13"} Nov 02 21:39:15.401189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:15.401100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" event={"ID":"7fdcfe05-bce2-46dd-9152-d8e08c669411","Type":"ContainerStarted","Data":"5ae7d74b854e602509e9186820f2a36d73fc38ae4a329cc5860c78a31500eb5b"} Nov 02 21:39:15.401189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:15.401167 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:15.538185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:15.538135 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" podStartSLOduration=2.869000114 podStartE2EDuration="7.53812015s" totalImagesPullingTime="4.118679435s" totalInitContainerRuntime="550.440601ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:15.536828129 +0000 UTC m=+302.698923249" watchObservedRunningTime="2026-11-02 21:39:15.53812015 +0000 UTC m=+302.700215273" Nov 02 21:39:18.538275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.538235 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk"] Nov 02 21:39:18.540732 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.540709 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:18.586764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.586735 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hbt\" (UniqueName: \"kubernetes.io/projected/186424fc-e8af-415d-8ed6-501e3640f01d-kube-api-access-k6hbt\") pod \"dns-operator-controller-manager-8557cbdb84-cn7qk\" (UID: \"186424fc-e8af-415d-8ed6-501e3640f01d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:18.643956 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.643926 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 21:39:18.643956 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.643935 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 21:39:18.644114 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.643982 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 21:39:18.644114 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.644029 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-szltp\"" Nov 02 21:39:18.645714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.645692 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk"] Nov 02 21:39:18.688177 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.688152 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hbt\" (UniqueName: \"kubernetes.io/projected/186424fc-e8af-415d-8ed6-501e3640f01d-kube-api-access-k6hbt\") pod \"dns-operator-controller-manager-8557cbdb84-cn7qk\" (UID: \"186424fc-e8af-415d-8ed6-501e3640f01d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:18.744794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.744767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hbt\" (UniqueName: \"kubernetes.io/projected/186424fc-e8af-415d-8ed6-501e3640f01d-kube-api-access-k6hbt\") pod \"dns-operator-controller-manager-8557cbdb84-cn7qk\" (UID: \"186424fc-e8af-415d-8ed6-501e3640f01d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:18.852533 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:18.852470 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:19.224166 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:19.224127 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk"] Nov 02 21:39:19.226573 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:19.226541 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186424fc_e8af_415d_8ed6_501e3640f01d.slice/crio-06e82dfab7a6077f19c88437e87fc6397a328545970b08551786324e8ac761c4 WatchSource:0}: Error finding container 06e82dfab7a6077f19c88437e87fc6397a328545970b08551786324e8ac761c4: Status 404 returned error can't find the container with id 06e82dfab7a6077f19c88437e87fc6397a328545970b08551786324e8ac761c4 Nov 02 21:39:19.229315 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:19.229299 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:39:19.420125 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:19.420091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" event={"ID":"186424fc-e8af-415d-8ed6-501e3640f01d","Type":"ContainerStarted","Data":"06e82dfab7a6077f19c88437e87fc6397a328545970b08551786324e8ac761c4"} Nov 02 21:39:23.439105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:23.439073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" event={"ID":"186424fc-e8af-415d-8ed6-501e3640f01d","Type":"ContainerStarted","Data":"94d6005881a315f6a1df40a5bb9ddcfa0139f6166b1d7bb0528e933576686e42"} Nov 02 21:39:23.439432 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:23.439193 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:23.579692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:23.579618 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" podStartSLOduration=1.449459969 podStartE2EDuration="5.579605909s" totalImagesPullingTime="4.13014594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:23.57883016 +0000 UTC m=+310.740925279" watchObservedRunningTime="2026-11-02 21:39:23.579605909 +0000 UTC m=+310.741701028" Nov 02 21:39:24.707603 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.707531 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t"] Nov 02 21:39:24.709841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.709823 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:24.814287 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.814255 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lcwpm\"" Nov 02 21:39:24.816038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.816015 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t"] Nov 02 21:39:24.839687 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.839655 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmd56\" (UniqueName: \"kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nvf2t\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:24.839798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.839704 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nvf2t\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:24.940925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.940895 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmd56\" (UniqueName: \"kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nvf2t\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:24.941089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.940932 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nvf2t\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:24.941264 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.941247 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nvf2t\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:24.997442 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:24.997366 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmd56\" (UniqueName: \"kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nvf2t\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:25.020346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:25.020323 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:25.489267 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:25.489229 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t"] Nov 02 21:39:25.495291 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:25.495253 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda635ce_fe32_475a_bc68_05607568bf35.slice/crio-5687c57a410cfb24eab96b90fdf423909091ef5e5c4d0081faad057a4cb0d7de WatchSource:0}: Error finding container 5687c57a410cfb24eab96b90fdf423909091ef5e5c4d0081faad057a4cb0d7de: Status 404 returned error can't find the container with id 5687c57a410cfb24eab96b90fdf423909091ef5e5c4d0081faad057a4cb0d7de Nov 02 21:39:26.409417 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:26.409369 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-9xbzv" Nov 02 21:39:26.456443 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:26.456410 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" event={"ID":"bda635ce-fe32-475a-bc68-05607568bf35","Type":"ContainerStarted","Data":"5687c57a410cfb24eab96b90fdf423909091ef5e5c4d0081faad057a4cb0d7de"} Nov 02 21:39:27.462182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.462145 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" event={"ID":"05845a1f-26ab-4af4-b0cf-02fcf83aa677","Type":"ContainerStarted","Data":"bebf6716c6d77ed605b017e96cd49f6f57e3578d4520170d09e9b141dcfbf61b"} Nov 02 21:39:27.580122 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.580071 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4tqhb" podStartSLOduration=1.486611918 podStartE2EDuration="28.580054037s" totalImagesPullingTime="27.093442119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:27.578320626 +0000 UTC m=+314.740415748" watchObservedRunningTime="2026-11-02 21:39:27.580054037 +0000 UTC m=+314.742149157" Nov 02 21:39:27.690486 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.690448 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n"] Nov 02 21:39:27.693165 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.693141 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:27.745376 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.745310 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qwvtk\"" Nov 02 21:39:27.752731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.752704 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n"] Nov 02 21:39:27.766868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.766843 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8hf\" (UniqueName: \"kubernetes.io/projected/3244c502-b53c-4193-9e0d-3507245016ca-kube-api-access-7p8hf\") pod \"limitador-operator-controller-manager-5778bb4449-ns97n\" (UID: \"3244c502-b53c-4193-9e0d-3507245016ca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:27.867460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.867428 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8hf\" (UniqueName: \"kubernetes.io/projected/3244c502-b53c-4193-9e0d-3507245016ca-kube-api-access-7p8hf\") pod \"limitador-operator-controller-manager-5778bb4449-ns97n\" (UID: \"3244c502-b53c-4193-9e0d-3507245016ca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:27.935715 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:27.935680 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8hf\" (UniqueName: \"kubernetes.io/projected/3244c502-b53c-4193-9e0d-3507245016ca-kube-api-access-7p8hf\") pod \"limitador-operator-controller-manager-5778bb4449-ns97n\" (UID: \"3244c502-b53c-4193-9e0d-3507245016ca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:28.008188 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:28.008122 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:28.382162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:28.382120 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n"] Nov 02 21:39:28.385853 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:28.385826 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3244c502_b53c_4193_9e0d_3507245016ca.slice/crio-03b2941fbbd2f1d6abff0f7b7fb17f48071bbac474b5854ba29312ea5d53f62e WatchSource:0}: Error finding container 03b2941fbbd2f1d6abff0f7b7fb17f48071bbac474b5854ba29312ea5d53f62e: Status 404 returned error can't find the container with id 03b2941fbbd2f1d6abff0f7b7fb17f48071bbac474b5854ba29312ea5d53f62e Nov 02 21:39:28.469166 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:28.469130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" event={"ID":"3244c502-b53c-4193-9e0d-3507245016ca","Type":"ContainerStarted","Data":"03b2941fbbd2f1d6abff0f7b7fb17f48071bbac474b5854ba29312ea5d53f62e"} Nov 02 21:39:31.212312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.212280 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw"] Nov 02 21:39:31.214928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.214907 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:31.272429 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.272401 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jcr5f\"" Nov 02 21:39:31.295845 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.295809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q7s9\" (UniqueName: \"kubernetes.io/projected/d31b49e2-0bd4-4d60-9094-92628656dfec-kube-api-access-6q7s9\") pod \"authorino-operator-7b7bb4bc77-nc4jw\" (UID: \"d31b49e2-0bd4-4d60-9094-92628656dfec\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:31.317290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.317257 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw"] Nov 02 21:39:31.396902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.396869 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7s9\" (UniqueName: \"kubernetes.io/projected/d31b49e2-0bd4-4d60-9094-92628656dfec-kube-api-access-6q7s9\") pod \"authorino-operator-7b7bb4bc77-nc4jw\" (UID: \"d31b49e2-0bd4-4d60-9094-92628656dfec\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:31.464649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.464618 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q7s9\" (UniqueName: \"kubernetes.io/projected/d31b49e2-0bd4-4d60-9094-92628656dfec-kube-api-access-6q7s9\") pod \"authorino-operator-7b7bb4bc77-nc4jw\" (UID: \"d31b49e2-0bd4-4d60-9094-92628656dfec\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:31.494584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.494539 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" event={"ID":"3244c502-b53c-4193-9e0d-3507245016ca","Type":"ContainerStarted","Data":"67e6bf0a4790ef41753d829066e05d279204abfe2519f2730f2b1a287662ab35"} Nov 02 21:39:31.494765 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.494680 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:31.527889 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.527856 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:31.611660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.610827 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" podStartSLOduration=1.928662438 podStartE2EDuration="4.610807427s" totalImagesPullingTime="2.682144989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:31.609186496 +0000 UTC m=+318.771281618" watchObservedRunningTime="2026-11-02 21:39:31.610807427 +0000 UTC m=+318.772902546" Nov 02 21:39:31.909494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:31.909466 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw"] Nov 02 21:39:31.912088 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:31.912055 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31b49e2_0bd4_4d60_9094_92628656dfec.slice/crio-b7ae4416aeede1c1a8c5b96ba56415e16e0b08e9c81d0194eef9330e485fda9f WatchSource:0}: Error finding container b7ae4416aeede1c1a8c5b96ba56415e16e0b08e9c81d0194eef9330e485fda9f: Status 404 returned error can't find the container with id b7ae4416aeede1c1a8c5b96ba56415e16e0b08e9c81d0194eef9330e485fda9f Nov 02 21:39:32.500232 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:32.500204 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" event={"ID":"d31b49e2-0bd4-4d60-9094-92628656dfec","Type":"ContainerStarted","Data":"b7ae4416aeede1c1a8c5b96ba56415e16e0b08e9c81d0194eef9330e485fda9f"} Nov 02 21:39:33.507353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:33.507316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" event={"ID":"bda635ce-fe32-475a-bc68-05607568bf35","Type":"ContainerStarted","Data":"2183b79f0a72f90258120ad8dc3145c8e730ece8bb4a29790820d6708aef3ebe"} Nov 02 21:39:33.507771 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:33.507383 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:33.723855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:33.723784 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" podStartSLOduration=2.75183554 podStartE2EDuration="9.723763406s" totalImagesPullingTime="6.971927866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:33.671602655 +0000 UTC m=+320.833697788" watchObservedRunningTime="2026-11-02 21:39:33.723763406 +0000 UTC m=+320.885858528" Nov 02 21:39:34.446410 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:34.446376 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cn7qk" Nov 02 21:39:35.517934 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:35.517894 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" event={"ID":"d31b49e2-0bd4-4d60-9094-92628656dfec","Type":"ContainerStarted","Data":"b9e5761e3c7dd9a557b9d41892a3cddef986df10cfff3acf8341f420576b7971"} Nov 02 21:39:35.518328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:35.518022 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:35.632123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:35.631927 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" podStartSLOduration=1.590436896 podStartE2EDuration="4.631909917s" totalImagesPullingTime="3.041473021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:35.630824976 +0000 UTC m=+322.792920096" watchObservedRunningTime="2026-11-02 21:39:35.631909917 +0000 UTC m=+322.794005036" Nov 02 21:39:37.111894 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:39:37.111859 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:39:42.019409 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.019375 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:39:42.023340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.023320 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.077862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.077819 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 21:39:42.080354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.080325 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:39:42.134381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.134352 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nzqml\"" Nov 02 21:39:42.186551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.186520 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file\") pod \"limitador-limitador-74dfcfb894-wnq5z\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.186729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.186578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcl79\" (UniqueName: \"kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79\") pod \"limitador-limitador-74dfcfb894-wnq5z\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.272091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.272022 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:39:42.287600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.287575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcl79\" (UniqueName: \"kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79\") pod \"limitador-limitador-74dfcfb894-wnq5z\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.287752 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.287729 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file\") pod \"limitador-limitador-74dfcfb894-wnq5z\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.288302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.288283 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file\") pod \"limitador-limitador-74dfcfb894-wnq5z\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.343908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.343874 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcl79\" (UniqueName: \"kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79\") pod \"limitador-limitador-74dfcfb894-wnq5z\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.503284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.503250 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ns97n" Nov 02 21:39:42.636019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.635992 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:42.814389 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.814357 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd"] Nov 02 21:39:42.819012 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.818991 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.888392 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.888305 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bxsh6\"" Nov 02 21:39:42.892527 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.892484 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd"] Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.892943 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.892994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893021 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893039 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f1cb9-13f8-43f9-9940-8ba271e9181f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893070 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893098 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkvb\" (UniqueName: \"kubernetes.io/projected/769f1cb9-13f8-43f9-9940-8ba271e9181f-kube-api-access-zrkvb\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893119 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893153 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.893321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.893191 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993578 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993544 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993578 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993581 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993622 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f1cb9-13f8-43f9-9940-8ba271e9181f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993665 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkvb\" (UniqueName: \"kubernetes.io/projected/769f1cb9-13f8-43f9-9940-8ba271e9181f-kube-api-access-zrkvb\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.993871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993848 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.993973 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.994098 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.994133 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.994298 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994361 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.994320 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.994506 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.994479 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f1cb9-13f8-43f9-9940-8ba271e9181f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.995993 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.995971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:42.996266 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:42.996246 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:43.020658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.020636 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:39:43.022071 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:43.022049 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d367cb_d2b3_4c8a_b3bb_98cf348e254c.slice/crio-5b51d4559d8671b573206254e7d52412c28c2f2cd0647b67c2d92f508245b4dd WatchSource:0}: Error finding container 5b51d4559d8671b573206254e7d52412c28c2f2cd0647b67c2d92f508245b4dd: Status 404 returned error can't find the container with id 5b51d4559d8671b573206254e7d52412c28c2f2cd0647b67c2d92f508245b4dd Nov 02 21:39:43.051523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.051494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkvb\" (UniqueName: \"kubernetes.io/projected/769f1cb9-13f8-43f9-9940-8ba271e9181f-kube-api-access-zrkvb\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:43.052230 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.052212 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f1cb9-13f8-43f9-9940-8ba271e9181f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd\" (UID: \"769f1cb9-13f8-43f9-9940-8ba271e9181f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:43.132431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.132400 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:43.557456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.557371 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" event={"ID":"29d367cb-d2b3-4c8a-b3bb-98cf348e254c","Type":"ContainerStarted","Data":"5b51d4559d8671b573206254e7d52412c28c2f2cd0647b67c2d92f508245b4dd"} Nov 02 21:39:43.566909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.566886 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd"] Nov 02 21:39:43.568957 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:39:43.568925 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769f1cb9_13f8_43f9_9940_8ba271e9181f.slice/crio-ca0bd98a58590b8c342af4459c937d00081cb80f0c56be0004f2dd8b52e78496 WatchSource:0}: Error finding container ca0bd98a58590b8c342af4459c937d00081cb80f0c56be0004f2dd8b52e78496: Status 404 returned error can't find the container with id ca0bd98a58590b8c342af4459c937d00081cb80f0c56be0004f2dd8b52e78496 Nov 02 21:39:43.571093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.571059 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:39:43.571187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.571124 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:39:43.571187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:43.571155 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:39:44.514229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:44.514196 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:39:44.564366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:44.564332 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" event={"ID":"769f1cb9-13f8-43f9-9940-8ba271e9181f","Type":"ContainerStarted","Data":"0a9e0f56cdccb888da0bf6fa44747af469c107f1a4409524d044b65a1e70d9fc"} Nov 02 21:39:44.564510 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:44.564388 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" event={"ID":"769f1cb9-13f8-43f9-9940-8ba271e9181f","Type":"ContainerStarted","Data":"ca0bd98a58590b8c342af4459c937d00081cb80f0c56be0004f2dd8b52e78496"} Nov 02 21:39:44.796848 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:44.796745 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" podStartSLOduration=2.7967292280000002 podStartE2EDuration="2.796729228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:44.796089195 +0000 UTC m=+331.958184317" watchObservedRunningTime="2026-11-02 21:39:44.796729228 +0000 UTC m=+331.958824344" Nov 02 21:39:45.132975 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.132945 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:45.138365 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.138341 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:45.570538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.570456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" event={"ID":"29d367cb-d2b3-4c8a-b3bb-98cf348e254c","Type":"ContainerStarted","Data":"903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223"} Nov 02 21:39:45.570908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.570755 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:39:45.570908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.570800 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:45.571624 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.571604 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd" Nov 02 21:39:45.684309 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:45.684258 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" podStartSLOduration=2.465423607 podStartE2EDuration="4.684245549s" totalImagesPullingTime="2.218821942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:45.682454078 +0000 UTC m=+332.844549200" watchObservedRunningTime="2026-11-02 21:39:45.684245549 +0000 UTC m=+332.846340730" Nov 02 21:39:46.524801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:46.524772 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nc4jw" Nov 02 21:39:56.576651 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:39:56.576619 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:40:00.887614 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:00.887578 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:40:00.888039 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:00.887779 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" podUID="29d367cb-d2b3-4c8a-b3bb-98cf348e254c" containerName="limitador" containerID="cri-o://903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223" gracePeriod=30 Nov 02 21:40:01.435199 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.435167 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:40:01.546780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.546700 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79\" (UniqueName: \"kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79\") pod \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " Nov 02 21:40:01.546920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.546800 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file\" (UniqueName: \"kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file\") pod \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\" (UID: \"29d367cb-d2b3-4c8a-b3bb-98cf348e254c\") " Nov 02 21:40:01.547154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.547129 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file" pod "29d367cb-d2b3-4c8a-b3bb-98cf348e254c" (UID: "29d367cb-d2b3-4c8a-b3bb-98cf348e254c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:01.549305 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.549267 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79" pod "29d367cb-d2b3-4c8a-b3bb-98cf348e254c" (UID: "29d367cb-d2b3-4c8a-b3bb-98cf348e254c"). InnerVolumeSpecName "kube-api-access-xcl79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:01.648051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.648023 5091 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-config-file\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:40:01.648051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.648051 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcl79\" (UniqueName: \"kubernetes.io/projected/29d367cb-d2b3-4c8a-b3bb-98cf348e254c-kube-api-access-xcl79\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:40:01.652039 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.652008 5091 generic.go:356] "Generic (PLEG): container finished" podID="29d367cb-d2b3-4c8a-b3bb-98cf348e254c" containerID="903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223" exitCode=0 Nov 02 21:40:01.652149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.652066 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" Nov 02 21:40:01.652149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.652092 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" event={"ID":"29d367cb-d2b3-4c8a-b3bb-98cf348e254c","Type":"ContainerDied","Data":"903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223"} Nov 02 21:40:01.652220 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.652164 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wnq5z" event={"ID":"29d367cb-d2b3-4c8a-b3bb-98cf348e254c","Type":"ContainerDied","Data":"5b51d4559d8671b573206254e7d52412c28c2f2cd0647b67c2d92f508245b4dd"} Nov 02 21:40:01.652220 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.652189 5091 scope.go:122] "RemoveContainer" containerID="903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223" Nov 02 21:40:01.661298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.661277 5091 scope.go:122] "RemoveContainer" containerID="903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223" Nov 02 21:40:01.661589 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:40:01.661569 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223\": container with ID starting with 903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223 not found: ID does not exist" containerID="903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223" Nov 02 21:40:01.661656 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.661598 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223"} err="failed to get container status \"903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223\": rpc error: code = NotFound desc = could not find container \"903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223\": container with ID starting with 903eabb719a021f1682cfb0f123c52165fcc2bbde3310c9c7565177312089223 not found: ID does not exist" Nov 02 21:40:01.819435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.819357 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:40:01.823225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:01.823196 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wnq5z"] Nov 02 21:40:02.976231 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:02.976196 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d367cb-d2b3-4c8a-b3bb-98cf348e254c" path="/var/lib/kubelet/pods/29d367cb-d2b3-4c8a-b3bb-98cf348e254c/volumes" Nov 02 21:40:03.360085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.360010 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx"] Nov 02 21:40:03.360400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.360386 5091 cpu_manager.go:414] "containerMap: removing container" podUID="29d367cb-d2b3-4c8a-b3bb-98cf348e254c" containerName="limitador" Nov 02 21:40:03.360452 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.360401 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d367cb-d2b3-4c8a-b3bb-98cf348e254c" containerName="limitador" Nov 02 21:40:03.360490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.360478 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d367cb-d2b3-4c8a-b3bb-98cf348e254c" containerName="limitador" Nov 02 21:40:03.363258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.363233 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.462794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.462753 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09299017-c6eb-4a8c-be23-a49a9f72358b-manager-config\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.462794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.462794 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09299017-c6eb-4a8c-be23-a49a9f72358b-metrics-certs\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.462976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.462813 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjw6n\" (UniqueName: \"kubernetes.io/projected/09299017-c6eb-4a8c-be23-a49a9f72358b-kube-api-access-hjw6n\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.462976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.462952 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09299017-c6eb-4a8c-be23-a49a9f72358b-cert\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.467775 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.467749 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:40:03.467923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.467808 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 21:40:03.467990 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.467809 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 21:40:03.470876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.470858 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 21:40:03.470958 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.470925 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-mkrf7\"" Nov 02 21:40:03.470958 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.470939 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 21:40:03.472756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.472738 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx"] Nov 02 21:40:03.563974 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.563944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09299017-c6eb-4a8c-be23-a49a9f72358b-cert\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.564129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.564014 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09299017-c6eb-4a8c-be23-a49a9f72358b-manager-config\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.564129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.564043 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09299017-c6eb-4a8c-be23-a49a9f72358b-metrics-certs\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.564129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.564062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjw6n\" (UniqueName: \"kubernetes.io/projected/09299017-c6eb-4a8c-be23-a49a9f72358b-kube-api-access-hjw6n\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.564936 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.564908 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09299017-c6eb-4a8c-be23-a49a9f72358b-manager-config\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.566552 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.566529 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09299017-c6eb-4a8c-be23-a49a9f72358b-metrics-certs\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.566652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.566582 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09299017-c6eb-4a8c-be23-a49a9f72358b-cert\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.620510 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.620482 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjw6n\" (UniqueName: \"kubernetes.io/projected/09299017-c6eb-4a8c-be23-a49a9f72358b-kube-api-access-hjw6n\") pod \"jobset-controller-manager-586764c7dd-vhnrx\" (UID: \"09299017-c6eb-4a8c-be23-a49a9f72358b\") " pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:03.674799 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:03.674771 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:04.047586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.047556 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx"] Nov 02 21:40:04.049351 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:40:04.049320 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09299017_c6eb_4a8c_be23_a49a9f72358b.slice/crio-e71d1f39acb331367406bf411ff40e0edca094cf6ee5061e0f0e053857586cbb WatchSource:0}: Error finding container e71d1f39acb331367406bf411ff40e0edca094cf6ee5061e0f0e053857586cbb: Status 404 returned error can't find the container with id e71d1f39acb331367406bf411ff40e0edca094cf6ee5061e0f0e053857586cbb Nov 02 21:40:04.443137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.443100 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tdhm"] Nov 02 21:40:04.483727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.483700 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:04.501506 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.501480 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tdhm"] Nov 02 21:40:04.535748 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.535722 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zdb7j\"" Nov 02 21:40:04.571880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.571849 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pwn\" (UniqueName: \"kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn\") pod \"authorino-57f7f8659f-2tdhm\" (UID: \"38e7a4bf-aa13-4f41-ad21-719234ab6748\") " pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:04.671317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.671287 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" event={"ID":"09299017-c6eb-4a8c-be23-a49a9f72358b","Type":"ContainerStarted","Data":"e71d1f39acb331367406bf411ff40e0edca094cf6ee5061e0f0e053857586cbb"} Nov 02 21:40:04.672561 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.672543 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pwn\" (UniqueName: \"kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn\") pod \"authorino-57f7f8659f-2tdhm\" (UID: \"38e7a4bf-aa13-4f41-ad21-719234ab6748\") " pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:04.728874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.728808 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pwn\" (UniqueName: \"kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn\") pod \"authorino-57f7f8659f-2tdhm\" (UID: \"38e7a4bf-aa13-4f41-ad21-719234ab6748\") " pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:04.820718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:04.820688 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:05.209573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:05.209546 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tdhm"] Nov 02 21:40:05.211041 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:40:05.211012 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38e7a4bf_aa13_4f41_ad21_719234ab6748.slice/crio-98d32e4d76eb736b28766ef59e28224670ae388402e3e0334a5ea69fce10fab7 WatchSource:0}: Error finding container 98d32e4d76eb736b28766ef59e28224670ae388402e3e0334a5ea69fce10fab7: Status 404 returned error can't find the container with id 98d32e4d76eb736b28766ef59e28224670ae388402e3e0334a5ea69fce10fab7 Nov 02 21:40:05.677896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:05.677862 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" event={"ID":"38e7a4bf-aa13-4f41-ad21-719234ab6748","Type":"ContainerStarted","Data":"98d32e4d76eb736b28766ef59e28224670ae388402e3e0334a5ea69fce10fab7"} Nov 02 21:40:05.745260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:05.745230 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t"] Nov 02 21:40:05.745476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:05.745451 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" podUID="bda635ce-fe32-475a-bc68-05607568bf35" containerName="manager" containerID="cri-o://2183b79f0a72f90258120ad8dc3145c8e730ece8bb4a29790820d6708aef3ebe" gracePeriod=10 Nov 02 21:40:06.687537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.687502 5091 generic.go:356] "Generic (PLEG): container finished" podID="bda635ce-fe32-475a-bc68-05607568bf35" containerID="2183b79f0a72f90258120ad8dc3145c8e730ece8bb4a29790820d6708aef3ebe" exitCode=0 Nov 02 21:40:06.687957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.687588 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" event={"ID":"bda635ce-fe32-475a-bc68-05607568bf35","Type":"ContainerDied","Data":"2183b79f0a72f90258120ad8dc3145c8e730ece8bb4a29790820d6708aef3ebe"} Nov 02 21:40:06.713260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.713209 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:40:06.787949 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.787927 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume\") pod \"bda635ce-fe32-475a-bc68-05607568bf35\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " Nov 02 21:40:06.788368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.788348 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume" pod "bda635ce-fe32-475a-bc68-05607568bf35" (UID: "bda635ce-fe32-475a-bc68-05607568bf35"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:06.888532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.888500 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56\" (UniqueName: \"kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56\") pod \"bda635ce-fe32-475a-bc68-05607568bf35\" (UID: \"bda635ce-fe32-475a-bc68-05607568bf35\") " Nov 02 21:40:06.888774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.888759 5091 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda635ce-fe32-475a-bc68-05607568bf35-extensions-socket-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:40:06.890920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.890895 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56" pod "bda635ce-fe32-475a-bc68-05607568bf35" (UID: "bda635ce-fe32-475a-bc68-05607568bf35"). InnerVolumeSpecName "kube-api-access-nmd56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:06.989838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:06.989814 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmd56\" (UniqueName: \"kubernetes.io/projected/bda635ce-fe32-475a-bc68-05607568bf35-kube-api-access-nmd56\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:40:07.116837 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:40:07.116792 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:40:07.696085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:07.696052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" event={"ID":"bda635ce-fe32-475a-bc68-05607568bf35","Type":"ContainerDied","Data":"5687c57a410cfb24eab96b90fdf423909091ef5e5c4d0081faad057a4cb0d7de"} Nov 02 21:40:07.696085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:07.696071 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t" Nov 02 21:40:07.696529 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:07.696116 5091 scope.go:122] "RemoveContainer" containerID="2183b79f0a72f90258120ad8dc3145c8e730ece8bb4a29790820d6708aef3ebe" Nov 02 21:40:07.909439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:07.909395 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t"] Nov 02 21:40:07.959435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:07.959365 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nvf2t"] Nov 02 21:40:08.702666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:08.702634 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" event={"ID":"09299017-c6eb-4a8c-be23-a49a9f72358b","Type":"ContainerStarted","Data":"c911a1969646e8712614fc3809c47f7bff6c1daafcfc819fa448f2de153ad4cf"} Nov 02 21:40:08.703111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:08.702728 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:08.705087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:08.705064 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" event={"ID":"38e7a4bf-aa13-4f41-ad21-719234ab6748","Type":"ContainerStarted","Data":"ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1"} Nov 02 21:40:08.816013 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:08.815959 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" podStartSLOduration=1.736090656 podStartE2EDuration="5.815945349s" totalImagesPullingTime="4.079854693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:08.81426541 +0000 UTC m=+355.976360531" watchObservedRunningTime="2026-11-02 21:40:08.815945349 +0000 UTC m=+355.978040468" Nov 02 21:40:08.925639 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:08.925576 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" podStartSLOduration=1.734405154 podStartE2EDuration="4.925559966s" totalImagesPullingTime="3.191154812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:08.924630623 +0000 UTC m=+356.086725745" watchObservedRunningTime="2026-11-02 21:40:08.925559966 +0000 UTC m=+356.087655091" Nov 02 21:40:08.976120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:08.976046 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda635ce-fe32-475a-bc68-05607568bf35" path="/var/lib/kubelet/pods/bda635ce-fe32-475a-bc68-05607568bf35/volumes" Nov 02 21:40:19.714424 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:19.714394 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-586764c7dd-vhnrx" Nov 02 21:40:28.657179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.657147 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2m4g4"] Nov 02 21:40:28.657562 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.657512 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bda635ce-fe32-475a-bc68-05607568bf35" containerName="manager" Nov 02 21:40:28.657562 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.657523 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda635ce-fe32-475a-bc68-05607568bf35" containerName="manager" Nov 02 21:40:28.657631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.657593 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda635ce-fe32-475a-bc68-05607568bf35" containerName="manager" Nov 02 21:40:28.659650 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.659629 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:40:28.716525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.716498 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2m4g4"] Nov 02 21:40:28.757426 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.757396 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsfc\" (UniqueName: \"kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc\") pod \"authorino-56c6c9ddd5-2m4g4\" (UID: \"e45f507d-a933-4f6c-873f-2b4fffcb6cf6\") " pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:40:28.858353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.858324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsfc\" (UniqueName: \"kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc\") pod \"authorino-56c6c9ddd5-2m4g4\" (UID: \"e45f507d-a933-4f6c-873f-2b4fffcb6cf6\") " pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:40:28.914525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.914451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsfc\" (UniqueName: \"kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc\") pod \"authorino-56c6c9ddd5-2m4g4\" (UID: \"e45f507d-a933-4f6c-873f-2b4fffcb6cf6\") " pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:40:28.969421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:28.969385 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:40:29.334229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:29.334204 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2m4g4"] Nov 02 21:40:29.336559 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:40:29.336531 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode45f507d_a933_4f6c_873f_2b4fffcb6cf6.slice/crio-830a0c9909f9bb5c8e85908cb400e3cb34802d52ed62d69e2f9adbf96ec6868f WatchSource:0}: Error finding container 830a0c9909f9bb5c8e85908cb400e3cb34802d52ed62d69e2f9adbf96ec6868f: Status 404 returned error can't find the container with id 830a0c9909f9bb5c8e85908cb400e3cb34802d52ed62d69e2f9adbf96ec6868f Nov 02 21:40:29.803090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:29.803054 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" event={"ID":"e45f507d-a933-4f6c-873f-2b4fffcb6cf6","Type":"ContainerStarted","Data":"830a0c9909f9bb5c8e85908cb400e3cb34802d52ed62d69e2f9adbf96ec6868f"} Nov 02 21:40:30.810032 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:30.809996 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" event={"ID":"e45f507d-a933-4f6c-873f-2b4fffcb6cf6","Type":"ContainerStarted","Data":"dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0"} Nov 02 21:40:30.921961 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:30.921901 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" podStartSLOduration=1.519214579 podStartE2EDuration="2.921888475s" totalImagesPullingTime="1.402673896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:30.920782265 +0000 UTC m=+378.082877398" watchObservedRunningTime="2026-11-02 21:40:30.921888475 +0000 UTC m=+378.083983594" Nov 02 21:40:30.944787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:30.944755 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tdhm"] Nov 02 21:40:30.944981 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:30.944941 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" podUID="38e7a4bf-aa13-4f41-ad21-719234ab6748" containerName="authorino" containerID="cri-o://ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1" gracePeriod=30 Nov 02 21:40:31.190356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.190294 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:31.277078 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.277046 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn\" (UniqueName: \"kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn\") pod \"38e7a4bf-aa13-4f41-ad21-719234ab6748\" (UID: \"38e7a4bf-aa13-4f41-ad21-719234ab6748\") " Nov 02 21:40:31.279094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.279063 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn" pod "38e7a4bf-aa13-4f41-ad21-719234ab6748" (UID: "38e7a4bf-aa13-4f41-ad21-719234ab6748"). InnerVolumeSpecName "kube-api-access-z6pwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:31.378136 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.378108 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pwn\" (UniqueName: \"kubernetes.io/projected/38e7a4bf-aa13-4f41-ad21-719234ab6748-kube-api-access-z6pwn\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:40:31.815950 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.815863 5091 generic.go:356] "Generic (PLEG): container finished" podID="38e7a4bf-aa13-4f41-ad21-719234ab6748" containerID="ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1" exitCode=0 Nov 02 21:40:31.815950 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.815911 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" Nov 02 21:40:31.815950 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.815940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" event={"ID":"38e7a4bf-aa13-4f41-ad21-719234ab6748","Type":"ContainerDied","Data":"ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1"} Nov 02 21:40:31.816444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.815985 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tdhm" event={"ID":"38e7a4bf-aa13-4f41-ad21-719234ab6748","Type":"ContainerDied","Data":"98d32e4d76eb736b28766ef59e28224670ae388402e3e0334a5ea69fce10fab7"} Nov 02 21:40:31.816444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.816004 5091 scope.go:122] "RemoveContainer" containerID="ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1" Nov 02 21:40:31.825619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.825600 5091 scope.go:122] "RemoveContainer" containerID="ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1" Nov 02 21:40:31.825894 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:40:31.825876 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1\": container with ID starting with ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1 not found: ID does not exist" containerID="ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1" Nov 02 21:40:31.825946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.825904 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1"} err="failed to get container status \"ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1\": rpc error: code = NotFound desc = could not find container \"ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1\": container with ID starting with ec68f520209f3856e0ea274072f7630088e418926498ff83d4163598621f09f1 not found: ID does not exist" Nov 02 21:40:31.978097 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.978062 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tdhm"] Nov 02 21:40:31.983423 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:31.983395 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tdhm"] Nov 02 21:40:32.974433 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:40:32.974404 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e7a4bf-aa13-4f41-ad21-719234ab6748" path="/var/lib/kubelet/pods/38e7a4bf-aa13-4f41-ad21-719234ab6748/volumes" Nov 02 21:40:37.111768 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:40:37.111722 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:41:03.397076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.396993 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj"] Nov 02 21:41:03.397551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.397428 5091 cpu_manager.go:414] "containerMap: removing container" podUID="38e7a4bf-aa13-4f41-ad21-719234ab6748" containerName="authorino" Nov 02 21:41:03.397551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.397441 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e7a4bf-aa13-4f41-ad21-719234ab6748" containerName="authorino" Nov 02 21:41:03.397551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.397527 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e7a4bf-aa13-4f41-ad21-719234ab6748" containerName="authorino" Nov 02 21:41:03.399583 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.399567 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.452164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.452134 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 21:41:03.452164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.452146 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 21:41:03.452333 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.452195 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 21:41:03.452333 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.452196 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 21:41:03.501945 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.501922 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 21:41:03.502235 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.502208 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-r4s5v\"" Nov 02 21:41:03.505236 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.505170 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj"] Nov 02 21:41:03.562553 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.562525 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.562706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.562572 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgpj\" (UniqueName: \"kubernetes.io/projected/2a821de6-21e0-4677-bb47-5e2863b4d4c5-kube-api-access-5xgpj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.562706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.562599 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.562878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.562775 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a821de6-21e0-4677-bb47-5e2863b4d4c5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.663645 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.663572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.663645 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.663626 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgpj\" (UniqueName: \"kubernetes.io/projected/2a821de6-21e0-4677-bb47-5e2863b4d4c5-kube-api-access-5xgpj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.664045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.663665 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.664045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.663779 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a821de6-21e0-4677-bb47-5e2863b4d4c5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.664045 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:03.663839 5091 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 21:41:03.664045 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:03.663914 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-proxy-source podName:2a821de6-21e0-4677-bb47-5e2863b4d4c5 nodeName:}" failed. No retries permitted until 2026-11-02 21:41:04.163897981 +0000 UTC m=+411.325993099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" (UID: "2a821de6-21e0-4677-bb47-5e2863b4d4c5") : configmap "usage-tenancy-proxy-source" not found Nov 02 21:41:03.664312 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.664295 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.666157 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.666139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a821de6-21e0-4677-bb47-5e2863b4d4c5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:03.721083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:03.721060 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgpj\" (UniqueName: \"kubernetes.io/projected/2a821de6-21e0-4677-bb47-5e2863b4d4c5-kube-api-access-5xgpj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:04.168162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:04.168131 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:04.168896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:04.168875 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2a821de6-21e0-4677-bb47-5e2863b4d4c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ss7kj\" (UID: \"2a821de6-21e0-4677-bb47-5e2863b4d4c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:04.311519 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:04.311490 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:04.682664 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:04.682640 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj"] Nov 02 21:41:04.684528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:41:04.684503 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a821de6_21e0_4677_bb47_5e2863b4d4c5.slice/crio-2c9928e1901d34cf467dbfb176917078f3f112628655f1518515099db597f041 WatchSource:0}: Error finding container 2c9928e1901d34cf467dbfb176917078f3f112628655f1518515099db597f041: Status 404 returned error can't find the container with id 2c9928e1901d34cf467dbfb176917078f3f112628655f1518515099db597f041 Nov 02 21:41:04.975075 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:04.974991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" event={"ID":"2a821de6-21e0-4677-bb47-5e2863b4d4c5","Type":"ContainerStarted","Data":"2c9928e1901d34cf467dbfb176917078f3f112628655f1518515099db597f041"} Nov 02 21:41:07.111724 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:07.111660 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:41:13.068700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.068643 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv"] Nov 02 21:41:13.073756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.073733 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.128053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.128006 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 21:41:13.130501 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.130469 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv"] Nov 02 21:41:13.148678 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.148643 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4z8\" (UniqueName: \"kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8\") pod \"llmisvc-controller-manager-869d5dcc5d-6r7vv\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.148838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.148783 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6r7vv\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.177400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.177373 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kn2lv\"" Nov 02 21:41:13.250137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.250104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6r7vv\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.250295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.250227 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4z8\" (UniqueName: \"kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8\") pod \"llmisvc-controller-manager-869d5dcc5d-6r7vv\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.252792 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.252753 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6r7vv\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.309088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.309024 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4z8\" (UniqueName: \"kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8\") pod \"llmisvc-controller-manager-869d5dcc5d-6r7vv\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.389606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.389573 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:13.769300 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:13.769268 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv"] Nov 02 21:41:13.771637 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:41:13.771603 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe0a60f6_6bc3_4395_93fb_fbf89ce626cc.slice/crio-42371c96bb85df811926f9a8afe11611d632ea8d8f62337924a2bc9d580b09e3 WatchSource:0}: Error finding container 42371c96bb85df811926f9a8afe11611d632ea8d8f62337924a2bc9d580b09e3: Status 404 returned error can't find the container with id 42371c96bb85df811926f9a8afe11611d632ea8d8f62337924a2bc9d580b09e3 Nov 02 21:41:14.031971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:14.031894 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" event={"ID":"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc","Type":"ContainerStarted","Data":"42371c96bb85df811926f9a8afe11611d632ea8d8f62337924a2bc9d580b09e3"} Nov 02 21:41:18.056556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:18.056509 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" event={"ID":"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc","Type":"ContainerStarted","Data":"de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb"} Nov 02 21:41:18.057002 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:18.056590 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:18.169707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:18.169626 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" podStartSLOduration=1.714508662 podStartE2EDuration="5.169605726s" totalImagesPullingTime="3.455097064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:18.167818595 +0000 UTC m=+425.329913725" watchObservedRunningTime="2026-11-02 21:41:18.169605726 +0000 UTC m=+425.331700847" Nov 02 21:41:19.062740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:19.062702 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" event={"ID":"2a821de6-21e0-4677-bb47-5e2863b4d4c5","Type":"ContainerStarted","Data":"8481b7efab166e2a714f3010a6d14a9cb5948f0cc316e1140108e4a03975174b"} Nov 02 21:41:19.063142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:19.062888 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:19.175745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:19.175663 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" podStartSLOduration=2.614698839 podStartE2EDuration="16.175647929s" totalImagesPullingTime="13.56094909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:19.174297117 +0000 UTC m=+426.336392231" watchObservedRunningTime="2026-11-02 21:41:19.175647929 +0000 UTC m=+426.337743049" Nov 02 21:41:30.074460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:30.074431 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ss7kj" Nov 02 21:41:37.113000 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:37.112958 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:41:47.729802 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:47.729758 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btgv4"] Nov 02 21:41:47.765998 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:47.765963 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:47.788616 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:47.788578 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btgv4"] Nov 02 21:41:47.853142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:47.853099 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bln\" (UniqueName: \"kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln\") pod \"authorino-6f6944bcdd-btgv4\" (UID: \"68f3078f-66cd-43ec-b900-05f8986f3c40\") " pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:47.954018 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:47.953984 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bln\" (UniqueName: \"kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln\") pod \"authorino-6f6944bcdd-btgv4\" (UID: \"68f3078f-66cd-43ec-b900-05f8986f3c40\") " pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:48.010140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.010072 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bln\" (UniqueName: \"kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln\") pod \"authorino-6f6944bcdd-btgv4\" (UID: \"68f3078f-66cd-43ec-b900-05f8986f3c40\") " pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:48.078116 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.078083 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:48.081262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.081242 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btgv4"] Nov 02 21:41:48.103952 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.103917 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bf68cdc-z7fjw"] Nov 02 21:41:48.107020 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.106998 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:48.156255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.156223 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2524t\" (UniqueName: \"kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t\") pod \"authorino-7bf68cdc-z7fjw\" (UID: \"161b8605-4353-4a07-a19f-4b5d8703fe6b\") " pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:48.163044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.163019 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bf68cdc-z7fjw"] Nov 02 21:41:48.256927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.256898 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2524t\" (UniqueName: \"kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t\") pod \"authorino-7bf68cdc-z7fjw\" (UID: \"161b8605-4353-4a07-a19f-4b5d8703fe6b\") " pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:48.312995 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.312924 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2524t\" (UniqueName: \"kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t\") pod \"authorino-7bf68cdc-z7fjw\" (UID: \"161b8605-4353-4a07-a19f-4b5d8703fe6b\") " pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:48.422412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.422378 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:48.447857 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.447802 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btgv4"] Nov 02 21:41:48.450865 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:41:48.450833 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68f3078f_66cd_43ec_b900_05f8986f3c40.slice/crio-94bd2f69c6e76bef2283e50ea52f2de2eea4f3e93d5c2d8ff1d565b54fe94fd9 WatchSource:0}: Error finding container 94bd2f69c6e76bef2283e50ea52f2de2eea4f3e93d5c2d8ff1d565b54fe94fd9: Status 404 returned error can't find the container with id 94bd2f69c6e76bef2283e50ea52f2de2eea4f3e93d5c2d8ff1d565b54fe94fd9 Nov 02 21:41:48.558229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.558194 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bf68cdc-z7fjw"] Nov 02 21:41:48.585046 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.584964 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d5758d5c7-qxn7l"] Nov 02 21:41:48.588022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.587999 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.644602 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.644227 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d5758d5c7-qxn7l"] Nov 02 21:41:48.645258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.645231 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 21:41:48.661327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.661283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert\") pod \"authorino-7d5758d5c7-qxn7l\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.661499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.661370 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwjj\" (UniqueName: \"kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj\") pod \"authorino-7d5758d5c7-qxn7l\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.762864 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.762821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert\") pod \"authorino-7d5758d5c7-qxn7l\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.763270 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.762880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwjj\" (UniqueName: \"kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj\") pod \"authorino-7d5758d5c7-qxn7l\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.765412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.765378 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert\") pod \"authorino-7d5758d5c7-qxn7l\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.801703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.801654 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bf68cdc-z7fjw"] Nov 02 21:41:48.802988 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:41:48.802952 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod161b8605_4353_4a07_a19f_4b5d8703fe6b.slice/crio-9cfcd56f2157f9539099e8f6eabddfb7fc0894d1fd883f9a8f127c6c13c040fa WatchSource:0}: Error finding container 9cfcd56f2157f9539099e8f6eabddfb7fc0894d1fd883f9a8f127c6c13c040fa: Status 404 returned error can't find the container with id 9cfcd56f2157f9539099e8f6eabddfb7fc0894d1fd883f9a8f127c6c13c040fa Nov 02 21:41:48.819257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.819232 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwjj\" (UniqueName: \"kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj\") pod \"authorino-7d5758d5c7-qxn7l\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:48.898771 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:48.898732 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:41:49.065464 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:49.065426 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:41:49.221769 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:49.221693 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" event={"ID":"68f3078f-66cd-43ec-b900-05f8986f3c40","Type":"ContainerStarted","Data":"94bd2f69c6e76bef2283e50ea52f2de2eea4f3e93d5c2d8ff1d565b54fe94fd9"} Nov 02 21:41:49.223126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:49.223100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" event={"ID":"161b8605-4353-4a07-a19f-4b5d8703fe6b","Type":"ContainerStarted","Data":"9cfcd56f2157f9539099e8f6eabddfb7fc0894d1fd883f9a8f127c6c13c040fa"} Nov 02 21:41:49.274705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:49.274655 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d5758d5c7-qxn7l"] Nov 02 21:41:49.275191 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:41:49.275159 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0cd7e5c_7925_437f_a2b5_6ac132a3a68f.slice/crio-90ff5e90873bc6c47fbb77f4523e1c7ab80fe65b559347fffe007b5b7fa8c9df WatchSource:0}: Error finding container 90ff5e90873bc6c47fbb77f4523e1c7ab80fe65b559347fffe007b5b7fa8c9df: Status 404 returned error can't find the container with id 90ff5e90873bc6c47fbb77f4523e1c7ab80fe65b559347fffe007b5b7fa8c9df Nov 02 21:41:50.229465 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.229361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" event={"ID":"161b8605-4353-4a07-a19f-4b5d8703fe6b","Type":"ContainerStarted","Data":"7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba"} Nov 02 21:41:50.229465 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.229384 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" podUID="161b8605-4353-4a07-a19f-4b5d8703fe6b" containerName="authorino" containerID="cri-o://7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba" gracePeriod=30 Nov 02 21:41:50.231206 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.231177 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" event={"ID":"68f3078f-66cd-43ec-b900-05f8986f3c40","Type":"ContainerStarted","Data":"7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2"} Nov 02 21:41:50.231431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.231275 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" podUID="68f3078f-66cd-43ec-b900-05f8986f3c40" containerName="authorino" containerID="cri-o://7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2" gracePeriod=30 Nov 02 21:41:50.232430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.232404 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" event={"ID":"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f","Type":"ContainerStarted","Data":"90ff5e90873bc6c47fbb77f4523e1c7ab80fe65b559347fffe007b5b7fa8c9df"} Nov 02 21:41:50.342450 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.342142 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" podStartSLOduration=1.107128961 podStartE2EDuration="2.342121935s" totalImagesPullingTime="1.234992974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:50.340403778 +0000 UTC m=+457.502498899" watchObservedRunningTime="2026-11-02 21:41:50.342121935 +0000 UTC m=+457.504217127" Nov 02 21:41:50.451026 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.450976 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" podStartSLOduration=1.930056378 podStartE2EDuration="3.450960877s" totalImagesPullingTime="1.520904499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:50.450514273 +0000 UTC m=+457.612609384" watchObservedRunningTime="2026-11-02 21:41:50.450960877 +0000 UTC m=+457.613055997" Nov 02 21:41:50.506014 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.505985 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:50.511329 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.511305 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:50.580051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.580027 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln\" (UniqueName: \"kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln\") pod \"68f3078f-66cd-43ec-b900-05f8986f3c40\" (UID: \"68f3078f-66cd-43ec-b900-05f8986f3c40\") " Nov 02 21:41:50.580144 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.580103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t\" (UniqueName: \"kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t\") pod \"161b8605-4353-4a07-a19f-4b5d8703fe6b\" (UID: \"161b8605-4353-4a07-a19f-4b5d8703fe6b\") " Nov 02 21:41:50.582076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.582040 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln" pod "68f3078f-66cd-43ec-b900-05f8986f3c40" (UID: "68f3078f-66cd-43ec-b900-05f8986f3c40"). InnerVolumeSpecName "kube-api-access-t6bln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:50.582644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.582624 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t" pod "161b8605-4353-4a07-a19f-4b5d8703fe6b" (UID: "161b8605-4353-4a07-a19f-4b5d8703fe6b"). InnerVolumeSpecName "kube-api-access-2524t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:50.681064 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.681028 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2524t\" (UniqueName: \"kubernetes.io/projected/161b8605-4353-4a07-a19f-4b5d8703fe6b-kube-api-access-2524t\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:50.681064 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:50.681056 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bln\" (UniqueName: \"kubernetes.io/projected/68f3078f-66cd-43ec-b900-05f8986f3c40-kube-api-access-t6bln\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:51.239587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.238784 5091 generic.go:356] "Generic (PLEG): container finished" podID="161b8605-4353-4a07-a19f-4b5d8703fe6b" containerID="7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba" exitCode=0 Nov 02 21:41:51.239587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.238893 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" event={"ID":"161b8605-4353-4a07-a19f-4b5d8703fe6b","Type":"ContainerDied","Data":"7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba"} Nov 02 21:41:51.239587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.238929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" event={"ID":"161b8605-4353-4a07-a19f-4b5d8703fe6b","Type":"ContainerDied","Data":"9cfcd56f2157f9539099e8f6eabddfb7fc0894d1fd883f9a8f127c6c13c040fa"} Nov 02 21:41:51.239587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.238950 5091 scope.go:122] "RemoveContainer" containerID="7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba" Nov 02 21:41:51.239587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.239091 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bf68cdc-z7fjw" Nov 02 21:41:51.246658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.246629 5091 generic.go:356] "Generic (PLEG): container finished" podID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerID="ec78dd3a1e8ccad850d45e5ddab65490c1547cbe9d3d9d0d8dfec3ede1e2abb4" exitCode=0 Nov 02 21:41:51.246932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.246751 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" event={"ID":"6d69c63c-9787-40b3-a692-6f723a96cefa","Type":"ContainerDied","Data":"ec78dd3a1e8ccad850d45e5ddab65490c1547cbe9d3d9d0d8dfec3ede1e2abb4"} Nov 02 21:41:51.247170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.247149 5091 scope.go:122] "RemoveContainer" containerID="ec78dd3a1e8ccad850d45e5ddab65490c1547cbe9d3d9d0d8dfec3ede1e2abb4" Nov 02 21:41:51.248447 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248330 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:41:51.248447 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248423 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:41:51.248561 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248477 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:41:51.249294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248780 5091 generic.go:356] "Generic (PLEG): container finished" podID="68f3078f-66cd-43ec-b900-05f8986f3c40" containerID="7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2" exitCode=0 Nov 02 21:41:51.249294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248824 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" Nov 02 21:41:51.249294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248857 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" event={"ID":"68f3078f-66cd-43ec-b900-05f8986f3c40","Type":"ContainerDied","Data":"7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2"} Nov 02 21:41:51.249294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.248886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-btgv4" event={"ID":"68f3078f-66cd-43ec-b900-05f8986f3c40","Type":"ContainerDied","Data":"94bd2f69c6e76bef2283e50ea52f2de2eea4f3e93d5c2d8ff1d565b54fe94fd9"} Nov 02 21:41:51.251224 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.251184 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" event={"ID":"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f","Type":"ContainerStarted","Data":"2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c"} Nov 02 21:41:51.257702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.257653 5091 scope.go:122] "RemoveContainer" containerID="7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba" Nov 02 21:41:51.258022 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:51.257987 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba\": container with ID starting with 7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba not found: ID does not exist" containerID="7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba" Nov 02 21:41:51.258147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.258033 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba"} err="failed to get container status \"7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba\": rpc error: code = NotFound desc = could not find container \"7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba\": container with ID starting with 7f7b65c7b5949aa0976f676d3434ae7f8b99551d639048a2ecd67899f82ce9ba not found: ID does not exist" Nov 02 21:41:51.258147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.258056 5091 scope.go:122] "RemoveContainer" containerID="7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2" Nov 02 21:41:51.268891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.268848 5091 scope.go:122] "RemoveContainer" containerID="7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2" Nov 02 21:41:51.269269 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:51.269162 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2\": container with ID starting with 7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2 not found: ID does not exist" containerID="7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2" Nov 02 21:41:51.269269 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.269199 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2"} err="failed to get container status \"7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2\": rpc error: code = NotFound desc = could not find container \"7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2\": container with ID starting with 7c2458818ce9dc56b4425ffb96e88eb6cc95fdcf5e73ec66a385d8e2253eb3f2 not found: ID does not exist" Nov 02 21:41:51.408817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.408761 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btgv4"] Nov 02 21:41:51.414304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.414273 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-btgv4"] Nov 02 21:41:51.522744 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.521054 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bf68cdc-z7fjw"] Nov 02 21:41:51.522917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.522760 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bf68cdc-z7fjw"] Nov 02 21:41:51.758692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.757104 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" podStartSLOduration=2.517638044 podStartE2EDuration="3.757080574s" totalImagesPullingTime="1.23944253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:51.75482011 +0000 UTC m=+458.916915231" watchObservedRunningTime="2026-11-02 21:41:51.757080574 +0000 UTC m=+458.919175695" Nov 02 21:41:51.804782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.804565 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2m4g4"] Nov 02 21:41:51.804935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:51.804847 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" podUID="e45f507d-a933-4f6c-873f-2b4fffcb6cf6" containerName="authorino" containerID="cri-o://dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0" gracePeriod=30 Nov 02 21:41:52.094289 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.094259 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:41:52.206111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.205858 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc\" (UniqueName: \"kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc\") pod \"e45f507d-a933-4f6c-873f-2b4fffcb6cf6\" (UID: \"e45f507d-a933-4f6c-873f-2b4fffcb6cf6\") " Nov 02 21:41:52.208280 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.208251 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc" pod "e45f507d-a933-4f6c-873f-2b4fffcb6cf6" (UID: "e45f507d-a933-4f6c-873f-2b4fffcb6cf6"). InnerVolumeSpecName "kube-api-access-zhsfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:52.257873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.257838 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" event={"ID":"6d69c63c-9787-40b3-a692-6f723a96cefa","Type":"ContainerStarted","Data":"b5b1a25ae083faab5d21d54d43db2e5514006f43e508d2ccd0527c8aabfd2e61"} Nov 02 21:41:52.260029 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.260005 5091 generic.go:356] "Generic (PLEG): container finished" podID="e45f507d-a933-4f6c-873f-2b4fffcb6cf6" containerID="dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0" exitCode=0 Nov 02 21:41:52.260139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.260046 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" Nov 02 21:41:52.260139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.260073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" event={"ID":"e45f507d-a933-4f6c-873f-2b4fffcb6cf6","Type":"ContainerDied","Data":"dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0"} Nov 02 21:41:52.260139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.260104 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2m4g4" event={"ID":"e45f507d-a933-4f6c-873f-2b4fffcb6cf6","Type":"ContainerDied","Data":"830a0c9909f9bb5c8e85908cb400e3cb34802d52ed62d69e2f9adbf96ec6868f"} Nov 02 21:41:52.260139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.260127 5091 scope.go:122] "RemoveContainer" containerID="dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0" Nov 02 21:41:52.294585 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.294560 5091 scope.go:122] "RemoveContainer" containerID="dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0" Nov 02 21:41:52.294997 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:41:52.294976 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0\": container with ID starting with dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0 not found: ID does not exist" containerID="dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0" Nov 02 21:41:52.295085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.295009 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0"} err="failed to get container status \"dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0\": rpc error: code = NotFound desc = could not find container \"dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0\": container with ID starting with dc467cb63f2b588d167b96b0e2f672c4982e9dc0e2823cd98d0b95f499da46e0 not found: ID does not exist" Nov 02 21:41:52.304234 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.304209 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:41:52.305513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.305460 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5wrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 02 21:41:52.305606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.305507 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 02 21:41:52.307004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.306980 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsfc\" (UniqueName: \"kubernetes.io/projected/e45f507d-a933-4f6c-873f-2b4fffcb6cf6-kube-api-access-zhsfc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:52.496565 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.496531 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2m4g4"] Nov 02 21:41:52.499998 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.499972 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2m4g4"] Nov 02 21:41:52.990527 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.990492 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161b8605-4353-4a07-a19f-4b5d8703fe6b" path="/var/lib/kubelet/pods/161b8605-4353-4a07-a19f-4b5d8703fe6b/volumes" Nov 02 21:41:52.990878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.990862 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f3078f-66cd-43ec-b900-05f8986f3c40" path="/var/lib/kubelet/pods/68f3078f-66cd-43ec-b900-05f8986f3c40/volumes" Nov 02 21:41:52.991155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:52.991143 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45f507d-a933-4f6c-873f-2b4fffcb6cf6" path="/var/lib/kubelet/pods/e45f507d-a933-4f6c-873f-2b4fffcb6cf6/volumes" Nov 02 21:41:53.304791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:53.304712 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5wrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 02 21:41:53.304791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:53.304766 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 02 21:41:54.303798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:54.303765 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5wrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 02 21:41:54.303976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:54.303817 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 02 21:41:55.156451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:55.156414 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:41:55.304549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:55.304517 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5wrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 02 21:41:55.304745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:55.304566 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 02 21:41:56.303954 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:56.303919 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5wrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 02 21:41:56.304379 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:56.303975 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 02 21:41:57.355279 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:57.355246 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:41:57.356311 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:41:57.356275 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:42:07.111240 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:42:07.111192 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:42:32.996235 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996160 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vggpx"] Nov 02 21:42:32.996595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996556 5091 cpu_manager.go:414] "containerMap: removing container" podUID="68f3078f-66cd-43ec-b900-05f8986f3c40" containerName="authorino" Nov 02 21:42:32.996595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996569 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f3078f-66cd-43ec-b900-05f8986f3c40" containerName="authorino" Nov 02 21:42:32.996688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996600 5091 cpu_manager.go:414] "containerMap: removing container" podUID="161b8605-4353-4a07-a19f-4b5d8703fe6b" containerName="authorino" Nov 02 21:42:32.996688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996605 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161b8605-4353-4a07-a19f-4b5d8703fe6b" containerName="authorino" Nov 02 21:42:32.996688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996617 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e45f507d-a933-4f6c-873f-2b4fffcb6cf6" containerName="authorino" Nov 02 21:42:32.996688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996623 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45f507d-a933-4f6c-873f-2b4fffcb6cf6" containerName="authorino" Nov 02 21:42:32.996841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996695 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45f507d-a933-4f6c-873f-2b4fffcb6cf6" containerName="authorino" Nov 02 21:42:32.996841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996710 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f3078f-66cd-43ec-b900-05f8986f3c40" containerName="authorino" Nov 02 21:42:32.996841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.996724 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="161b8605-4353-4a07-a19f-4b5d8703fe6b" containerName="authorino" Nov 02 21:42:32.998496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:32.998480 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.042839 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.042803 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a68a0a17-418b-4163-9d20-42f126791bb8-cert\") pod \"llmisvc-controller-manager-6466c554d8-vggpx\" (UID: \"a68a0a17-418b-4163-9d20-42f126791bb8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.042979 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.042850 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbv9\" (UniqueName: \"kubernetes.io/projected/a68a0a17-418b-4163-9d20-42f126791bb8-kube-api-access-fpbv9\") pod \"llmisvc-controller-manager-6466c554d8-vggpx\" (UID: \"a68a0a17-418b-4163-9d20-42f126791bb8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.056915 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.056887 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vggpx"] Nov 02 21:42:33.143640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.143606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a68a0a17-418b-4163-9d20-42f126791bb8-cert\") pod \"llmisvc-controller-manager-6466c554d8-vggpx\" (UID: \"a68a0a17-418b-4163-9d20-42f126791bb8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.143640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.143643 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbv9\" (UniqueName: \"kubernetes.io/projected/a68a0a17-418b-4163-9d20-42f126791bb8-kube-api-access-fpbv9\") pod \"llmisvc-controller-manager-6466c554d8-vggpx\" (UID: \"a68a0a17-418b-4163-9d20-42f126791bb8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.146072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.146050 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a68a0a17-418b-4163-9d20-42f126791bb8-cert\") pod \"llmisvc-controller-manager-6466c554d8-vggpx\" (UID: \"a68a0a17-418b-4163-9d20-42f126791bb8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.199953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.199921 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbv9\" (UniqueName: \"kubernetes.io/projected/a68a0a17-418b-4163-9d20-42f126791bb8-kube-api-access-fpbv9\") pod \"llmisvc-controller-manager-6466c554d8-vggpx\" (UID: \"a68a0a17-418b-4163-9d20-42f126791bb8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.310191 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.310109 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:33.678287 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:33.678259 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vggpx"] Nov 02 21:42:33.680484 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:42:33.680456 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68a0a17_418b_4163_9d20_42f126791bb8.slice/crio-7d99df9e74a96a0cb10344554d1ea50384e7553fe35958d341051850c56af88f WatchSource:0}: Error finding container 7d99df9e74a96a0cb10344554d1ea50384e7553fe35958d341051850c56af88f: Status 404 returned error can't find the container with id 7d99df9e74a96a0cb10344554d1ea50384e7553fe35958d341051850c56af88f Nov 02 21:42:34.500133 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:34.500097 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" event={"ID":"a68a0a17-418b-4163-9d20-42f126791bb8","Type":"ContainerStarted","Data":"6335040c23c271365c06ca4a588f985fe5cb0855d11203d23c559155104415c4"} Nov 02 21:42:34.500475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:34.500193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" event={"ID":"a68a0a17-418b-4163-9d20-42f126791bb8","Type":"ContainerStarted","Data":"7d99df9e74a96a0cb10344554d1ea50384e7553fe35958d341051850c56af88f"} Nov 02 21:42:34.500475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:34.500219 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:42:34.614742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:42:34.614691 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" podStartSLOduration=1.859759805 podStartE2EDuration="2.614658964s" totalImagesPullingTime="754.899159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:42:34.613016511 +0000 UTC m=+501.775111623" watchObservedRunningTime="2026-11-02 21:42:34.614658964 +0000 UTC m=+501.776754085" Nov 02 21:42:37.111163 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:42:37.111113 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:43:05.506414 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:05.506381 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vggpx" Nov 02 21:43:05.671980 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:05.671943 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv"] Nov 02 21:43:05.672251 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:05.672223 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" podUID="fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" containerName="manager" containerID="cri-o://de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb" gracePeriod=30 Nov 02 21:43:05.918141 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:05.918112 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:43:06.022537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.022476 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8\" (UniqueName: \"kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8\") pod \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " Nov 02 21:43:06.022649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.022546 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert\" (UniqueName: \"kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert\") pod \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\" (UID: \"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc\") " Nov 02 21:43:06.024645 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.024613 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert" pod "fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" (UID: "fe0a60f6-6bc3-4395-93fb-fbf89ce626cc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:06.024791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.024765 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8" pod "fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" (UID: "fe0a60f6-6bc3-4395-93fb-fbf89ce626cc"). InnerVolumeSpecName "kube-api-access-rz4z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:06.124101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.124068 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4z8\" (UniqueName: \"kubernetes.io/projected/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-kube-api-access-rz4z8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:43:06.124101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.124098 5091 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:43:06.654911 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.654874 5091 generic.go:356] "Generic (PLEG): container finished" podID="fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" containerID="de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb" exitCode=0 Nov 02 21:43:06.655368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.654949 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" Nov 02 21:43:06.655368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.654968 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" event={"ID":"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc","Type":"ContainerDied","Data":"de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb"} Nov 02 21:43:06.655368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.655070 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv" event={"ID":"fe0a60f6-6bc3-4395-93fb-fbf89ce626cc","Type":"ContainerDied","Data":"42371c96bb85df811926f9a8afe11611d632ea8d8f62337924a2bc9d580b09e3"} Nov 02 21:43:06.655368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.655104 5091 scope.go:122] "RemoveContainer" containerID="de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb" Nov 02 21:43:06.664194 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.664176 5091 scope.go:122] "RemoveContainer" containerID="de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb" Nov 02 21:43:06.664513 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:43:06.664487 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb\": container with ID starting with de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb not found: ID does not exist" containerID="de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb" Nov 02 21:43:06.664630 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.664524 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb"} err="failed to get container status \"de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb\": rpc error: code = NotFound desc = could not find container \"de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb\": container with ID starting with de98ced430ecd468a3127cfe9f71350c5d5eb220522b476320cec45b68c8d5cb not found: ID does not exist" Nov 02 21:43:06.819680 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.819635 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv"] Nov 02 21:43:06.869398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.869367 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6r7vv"] Nov 02 21:43:06.975597 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:43:06.975530 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" path="/var/lib/kubelet/pods/fe0a60f6-6bc3-4395-93fb-fbf89ce626cc/volumes" Nov 02 21:43:07.111538 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:43:07.111502 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:43:37.114407 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:43:37.114361 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:44:07.111295 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:44:07.111256 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:44:13.043454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:13.043427 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:44:13.044184 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:13.044165 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:44:18.725086 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.725052 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm"] Nov 02 21:44:18.725544 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.725447 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" containerName="manager" Nov 02 21:44:18.725544 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.725460 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" containerName="manager" Nov 02 21:44:18.725623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.725558 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0a60f6-6bc3-4395-93fb-fbf89ce626cc" containerName="manager" Nov 02 21:44:18.728916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.728897 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.832817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.832784 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm"] Nov 02 21:44:18.839325 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839294 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839330 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839412 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839452 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbdf6bd-a942-433c-a08e-ca09206efe34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839523 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xks\" (UniqueName: \"kubernetes.io/projected/ebbdf6bd-a942-433c-a08e-ca09206efe34-kube-api-access-v9xks\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839556 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839621 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.839879 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.839651 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940602 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940643 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbdf6bd-a942-433c-a08e-ca09206efe34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940665 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xks\" (UniqueName: \"kubernetes.io/projected/ebbdf6bd-a942-433c-a08e-ca09206efe34-kube-api-access-v9xks\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940703 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940721 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940747 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940766 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.940806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.941120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.940820 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.941179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.941148 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.941237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.941208 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.941294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.941253 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.941339 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.941303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.941503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.941482 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbdf6bd-a942-433c-a08e-ca09206efe34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.943120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.943096 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.943269 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.943248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.996729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.996663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbdf6bd-a942-433c-a08e-ca09206efe34-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:18.996855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:18.996835 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xks\" (UniqueName: \"kubernetes.io/projected/ebbdf6bd-a942-433c-a08e-ca09206efe34-kube-api-access-v9xks\") pod \"maas-default-gateway-openshift-default-65c448b547-m2sxm\" (UID: \"ebbdf6bd-a942-433c-a08e-ca09206efe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:19.041595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:19.041572 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:19.463183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:19.463158 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm"] Nov 02 21:44:19.464968 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:44:19.464942 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbdf6bd_a942_433c_a08e_ca09206efe34.slice/crio-1a32d9566c749c87ebaaa0d38fa97d7b29e10df96a49391462500d1f6e85dba6 WatchSource:0}: Error finding container 1a32d9566c749c87ebaaa0d38fa97d7b29e10df96a49391462500d1f6e85dba6: Status 404 returned error can't find the container with id 1a32d9566c749c87ebaaa0d38fa97d7b29e10df96a49391462500d1f6e85dba6 Nov 02 21:44:19.466959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:19.466927 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:19.467026 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:19.467014 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:19.467062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:19.467045 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:20.031806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:20.031770 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" event={"ID":"ebbdf6bd-a942-433c-a08e-ca09206efe34","Type":"ContainerStarted","Data":"06f89f17b48542c83721735cef6caa4ae688d18338aff89af947f3d9782b61e4"} Nov 02 21:44:20.032244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:20.031823 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" event={"ID":"ebbdf6bd-a942-433c-a08e-ca09206efe34","Type":"ContainerStarted","Data":"1a32d9566c749c87ebaaa0d38fa97d7b29e10df96a49391462500d1f6e85dba6"} Nov 02 21:44:20.042140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:20.042119 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:20.043554 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:20.043532 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:20.043642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:20.043573 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:20.151085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:20.151030 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podStartSLOduration=2.151012893 podStartE2EDuration="2.151012893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:20.149273654 +0000 UTC m=+607.311368775" watchObservedRunningTime="2026-11-02 21:44:20.151012893 +0000 UTC m=+607.313108015" Nov 02 21:44:21.014116 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.014081 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-555867774b-47zn5"] Nov 02 21:44:21.017686 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.017656 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.042569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.042539 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:21.042946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.042591 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:21.092062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.092031 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-555867774b-47zn5"] Nov 02 21:44:21.160271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.160217 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8649ce6-c528-455c-81b4-183e5012d436-tls-cert\") pod \"authorino-555867774b-47zn5\" (UID: \"d8649ce6-c528-455c-81b4-183e5012d436\") " pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.160579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.160551 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lzk\" (UniqueName: \"kubernetes.io/projected/d8649ce6-c528-455c-81b4-183e5012d436-kube-api-access-n8lzk\") pod \"authorino-555867774b-47zn5\" (UID: \"d8649ce6-c528-455c-81b4-183e5012d436\") " pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.261814 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.261771 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lzk\" (UniqueName: \"kubernetes.io/projected/d8649ce6-c528-455c-81b4-183e5012d436-kube-api-access-n8lzk\") pod \"authorino-555867774b-47zn5\" (UID: \"d8649ce6-c528-455c-81b4-183e5012d436\") " pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.261980 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.261849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8649ce6-c528-455c-81b4-183e5012d436-tls-cert\") pod \"authorino-555867774b-47zn5\" (UID: \"d8649ce6-c528-455c-81b4-183e5012d436\") " pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.288641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.288577 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8649ce6-c528-455c-81b4-183e5012d436-tls-cert\") pod \"authorino-555867774b-47zn5\" (UID: \"d8649ce6-c528-455c-81b4-183e5012d436\") " pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.320194 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.320167 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lzk\" (UniqueName: \"kubernetes.io/projected/d8649ce6-c528-455c-81b4-183e5012d436-kube-api-access-n8lzk\") pod \"authorino-555867774b-47zn5\" (UID: \"d8649ce6-c528-455c-81b4-183e5012d436\") " pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.330000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.329976 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555867774b-47zn5" Nov 02 21:44:21.719362 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.719335 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-555867774b-47zn5"] Nov 02 21:44:21.721769 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:44:21.721732 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8649ce6_c528_455c_81b4_183e5012d436.slice/crio-d55090705d1ef8d4599b049535e802b5f28098594d79b02210b22252611640f9 WatchSource:0}: Error finding container d55090705d1ef8d4599b049535e802b5f28098594d79b02210b22252611640f9: Status 404 returned error can't find the container with id d55090705d1ef8d4599b049535e802b5f28098594d79b02210b22252611640f9 Nov 02 21:44:21.723564 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:21.723538 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:44:22.041983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:22.041885 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:22.041983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:22.041923 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555867774b-47zn5" event={"ID":"d8649ce6-c528-455c-81b4-183e5012d436","Type":"ContainerStarted","Data":"d55090705d1ef8d4599b049535e802b5f28098594d79b02210b22252611640f9"} Nov 02 21:44:22.041983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:22.041938 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:23.042209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:23.042172 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:23.042766 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:23.042228 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:24.042434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.042395 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:24.042866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.042458 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:24.052325 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.052298 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555867774b-47zn5" event={"ID":"d8649ce6-c528-455c-81b4-183e5012d436","Type":"ContainerStarted","Data":"7cd3c3654a8aa686e949fc03bf0023e8ceec0eff781b7c49db0958ac10bbbd79"} Nov 02 21:44:24.187745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.167524 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-555867774b-47zn5" podStartSLOduration=2.373716345 podStartE2EDuration="4.167503827s" totalImagesPullingTime="1.793787482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:24.162468901 +0000 UTC m=+611.324564021" watchObservedRunningTime="2026-11-02 21:44:24.167503827 +0000 UTC m=+611.329598949" Nov 02 21:44:24.205215 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.205172 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d5758d5c7-qxn7l"] Nov 02 21:44:24.205409 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.205372 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" podUID="f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" containerName="authorino" containerID="cri-o://2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c" gracePeriod=30 Nov 02 21:44:24.506766 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.506741 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:44:24.616736 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.616640 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert\" (UniqueName: \"kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert\") pod \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " Nov 02 21:44:24.616736 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.616711 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj\" (UniqueName: \"kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj\") pod \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\" (UID: \"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f\") " Nov 02 21:44:24.618893 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.618850 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj" pod "f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" (UID: "f0cd7e5c-7925-437f-a2b5-6ac132a3a68f"). InnerVolumeSpecName "kube-api-access-zlwjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:24.632439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.632411 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert" pod "f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" (UID: "f0cd7e5c-7925-437f-a2b5-6ac132a3a68f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:24.717284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.717246 5091 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-tls-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:44:24.717284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:24.717276 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlwjj\" (UniqueName: \"kubernetes.io/projected/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f-kube-api-access-zlwjj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:44:25.042485 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.042449 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:25.042971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.042514 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:25.058812 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.058779 5091 generic.go:356] "Generic (PLEG): container finished" podID="f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" containerID="2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c" exitCode=0 Nov 02 21:44:25.058957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.058822 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" Nov 02 21:44:25.058957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.058867 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" event={"ID":"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f","Type":"ContainerDied","Data":"2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c"} Nov 02 21:44:25.058957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.058907 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5758d5c7-qxn7l" event={"ID":"f0cd7e5c-7925-437f-a2b5-6ac132a3a68f","Type":"ContainerDied","Data":"90ff5e90873bc6c47fbb77f4523e1c7ab80fe65b559347fffe007b5b7fa8c9df"} Nov 02 21:44:25.058957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.058928 5091 scope.go:122] "RemoveContainer" containerID="2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c" Nov 02 21:44:25.095809 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.095783 5091 scope.go:122] "RemoveContainer" containerID="2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c" Nov 02 21:44:25.096146 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:44:25.096127 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c\": container with ID starting with 2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c not found: ID does not exist" containerID="2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c" Nov 02 21:44:25.096213 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.096157 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c"} err="failed to get container status \"2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c\": rpc error: code = NotFound desc = could not find container \"2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c\": container with ID starting with 2872a1d783397d4b0abcd8032b4bdc713da9ca80e779d13a9ad8caeed747c70c not found: ID does not exist" Nov 02 21:44:25.222521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.222484 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d5758d5c7-qxn7l"] Nov 02 21:44:25.226111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:25.226082 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d5758d5c7-qxn7l"] Nov 02 21:44:26.042404 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:26.042351 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:44:26.042605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:26.042421 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:44:26.976696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:26.976631 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" path="/var/lib/kubelet/pods/f0cd7e5c-7925-437f-a2b5-6ac132a3a68f/volumes" Nov 02 21:44:27.826763 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:27.826715 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:27.827239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:27.827111 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:27.827930 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:44:27.827904 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" Nov 02 21:44:37.111110 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:44:37.111071 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:45:04.092871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.092836 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm"] Nov 02 21:45:04.093343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.093247 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" containerName="authorino" Nov 02 21:45:04.093343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.093259 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" containerName="authorino" Nov 02 21:45:04.093343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.093316 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cd7e5c-7925-437f-a2b5-6ac132a3a68f" containerName="authorino" Nov 02 21:45:04.096357 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.096339 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.182620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.182585 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b93f41f-2956-4052-a31c-ae1aded9a235-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.182782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.182629 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.182782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.182722 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.182782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.182762 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.182944 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.182890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp742\" (UniqueName: \"kubernetes.io/projected/5b93f41f-2956-4052-a31c-ae1aded9a235-kube-api-access-tp742\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.183002 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.182978 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.183058 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.183012 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.183058 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.183042 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.183129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.183116 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.283724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.283663 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.283905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.283736 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.283905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.283766 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.283905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.283791 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.283905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.283822 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b93f41f-2956-4052-a31c-ae1aded9a235-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.283950 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284007 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284046 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284115 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp742\" (UniqueName: \"kubernetes.io/projected/5b93f41f-2956-4052-a31c-ae1aded9a235-kube-api-access-tp742\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284249 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284127 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284318 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284258 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284325 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284347 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.284693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.284648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b93f41f-2956-4052-a31c-ae1aded9a235-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.286347 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.286193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.286787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.286768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.301105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.301023 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm"] Nov 02 21:45:04.340051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.340027 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b93f41f-2956-4052-a31c-ae1aded9a235-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.340259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.340238 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp742\" (UniqueName: \"kubernetes.io/projected/5b93f41f-2956-4052-a31c-ae1aded9a235-kube-api-access-tp742\") pod \"maas-default-gateway-openshift-default-65c448b547-xbnfm\" (UID: \"5b93f41f-2956-4052-a31c-ae1aded9a235\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.408228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.408201 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:04.833202 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.833125 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm"] Nov 02 21:45:04.870240 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:04.870209 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b93f41f_2956_4052_a31c_ae1aded9a235.slice/crio-587f5963c8caecc37bfdfe9be2eb8c82a67b659dcd65446b283baf2e9f55706d WatchSource:0}: Error finding container 587f5963c8caecc37bfdfe9be2eb8c82a67b659dcd65446b283baf2e9f55706d: Status 404 returned error can't find the container with id 587f5963c8caecc37bfdfe9be2eb8c82a67b659dcd65446b283baf2e9f55706d Nov 02 21:45:04.872188 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.872157 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.872281 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.872219 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.872281 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:04.872247 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:05.285183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:05.284647 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" event={"ID":"5b93f41f-2956-4052-a31c-ae1aded9a235","Type":"ContainerStarted","Data":"39a4c96a13f51131e065b5ed08867416af1a3b15ecd3c699daffe595c2e56ff3"} Nov 02 21:45:05.285183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:05.284766 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" event={"ID":"5b93f41f-2956-4052-a31c-ae1aded9a235","Type":"ContainerStarted","Data":"587f5963c8caecc37bfdfe9be2eb8c82a67b659dcd65446b283baf2e9f55706d"} Nov 02 21:45:05.402416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:05.402362 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podStartSLOduration=1.402345457 podStartE2EDuration="1.402345457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:05.400890252 +0000 UTC m=+652.562985372" watchObservedRunningTime="2026-11-02 21:45:05.402345457 +0000 UTC m=+652.564440578" Nov 02 21:45:05.408589 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:05.408557 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:05.409696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:05.409650 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:05.409813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:05.409707 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:06.409573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:06.409539 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:06.410192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:06.410160 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:07.114440 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:45:07.114389 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:45:07.409302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:07.409265 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:07.409472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:07.409336 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:08.409327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:08.409288 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:08.409866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:08.409371 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:09.409147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:09.409102 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:09.409331 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:09.409170 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:10.409040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:10.409001 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:10.409444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:10.409072 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:11.409017 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:11.408980 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 02 21:45:11.409193 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:11.409046 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 02 21:45:13.071782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:13.071744 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:13.072472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:13.072405 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:13.073371 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:13.073342 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" Nov 02 21:45:32.093277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.093194 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f"] Nov 02 21:45:32.097022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.097005 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.131566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.131539 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsm5\" (UniqueName: \"kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5\") pod \"payload-processing-e2e-worker-w3-64438f-cb5564875-z785f\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.131705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.131579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-64438f-cb5564875-z785f\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.200542 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.200490 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-64438f\"" Nov 02 21:45:32.200542 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.200498 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-64438f-dockercfg-hrx4n\"" Nov 02 21:45:32.202435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.202414 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f"] Nov 02 21:45:32.233107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.233084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsm5\" (UniqueName: \"kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5\") pod \"payload-processing-e2e-worker-w3-64438f-cb5564875-z785f\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.233219 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.233120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-64438f-cb5564875-z785f\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.233827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.233809 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-64438f-cb5564875-z785f\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.289627 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.289596 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsm5\" (UniqueName: \"kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5\") pod \"payload-processing-e2e-worker-w3-64438f-cb5564875-z785f\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.410606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.410572 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:32.778955 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:32.778931 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f"] Nov 02 21:45:32.781181 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:32.781150 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265d1fb0_e6b0_4964_88f9_bdb1007447f5.slice/crio-a3444796168f2d92518524b3ac7fd022ca1b79e21a86da3ae4af50cb7911c8ab WatchSource:0}: Error finding container a3444796168f2d92518524b3ac7fd022ca1b79e21a86da3ae4af50cb7911c8ab: Status 404 returned error can't find the container with id a3444796168f2d92518524b3ac7fd022ca1b79e21a86da3ae4af50cb7911c8ab Nov 02 21:45:33.436038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:33.436006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" event={"ID":"265d1fb0-e6b0-4964-88f9-bdb1007447f5","Type":"ContainerStarted","Data":"a3444796168f2d92518524b3ac7fd022ca1b79e21a86da3ae4af50cb7911c8ab"} Nov 02 21:45:36.454292 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.454258 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" event={"ID":"265d1fb0-e6b0-4964-88f9-bdb1007447f5","Type":"ContainerStarted","Data":"46407fd0f8a8bd4deeed4c62a6d10ea625fa4f8c655464d729545ec65e60b3ba"} Nov 02 21:45:36.454721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.454396 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:36.571491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.571443 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" podStartSLOduration=1.947857234 podStartE2EDuration="4.571430371s" totalImagesPullingTime="2.623573137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:36.569513557 +0000 UTC m=+683.731608676" watchObservedRunningTime="2026-11-02 21:45:36.571430371 +0000 UTC m=+683.733525490" Nov 02 21:45:36.717450 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.717372 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh"] Nov 02 21:45:36.720935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.720920 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:36.775664 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.775633 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fb8112\"" Nov 02 21:45:36.776194 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.776165 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-fb8112-dockercfg-kpfmk\"" Nov 02 21:45:36.778985 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.778952 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh"] Nov 02 21:45:36.779460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.779420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:36.779579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.779497 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49c7\" (UniqueName: \"kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7\") pod \"payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:36.880254 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.880212 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49c7\" (UniqueName: \"kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7\") pod \"payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:36.880440 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.880338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:36.881080 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.881053 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:36.936359 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:36.936326 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49c7\" (UniqueName: \"kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7\") pod \"payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:37.033831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:37.033767 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:37.117802 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:45:37.117767 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:45:37.405074 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:37.405047 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh"] Nov 02 21:45:37.406982 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:37.406948 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafc7e1c_50b3_4018_81fa_3da2fc5c04f1.slice/crio-d3b2fb9eb9801df4d8092d87ee3fd480868da24e0bde5d73f63aa9472b56a297 WatchSource:0}: Error finding container d3b2fb9eb9801df4d8092d87ee3fd480868da24e0bde5d73f63aa9472b56a297: Status 404 returned error can't find the container with id d3b2fb9eb9801df4d8092d87ee3fd480868da24e0bde5d73f63aa9472b56a297 Nov 02 21:45:37.461377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:37.461351 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" event={"ID":"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1","Type":"ContainerStarted","Data":"d3b2fb9eb9801df4d8092d87ee3fd480868da24e0bde5d73f63aa9472b56a297"} Nov 02 21:45:38.468087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:38.468057 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" event={"ID":"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1","Type":"ContainerStarted","Data":"f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff"} Nov 02 21:45:38.468531 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:38.468164 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:38.580098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:38.580052 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" podStartSLOduration=2.580039044 podStartE2EDuration="2.580039044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:38.579074961 +0000 UTC m=+685.741170072" watchObservedRunningTime="2026-11-02 21:45:38.580039044 +0000 UTC m=+685.742134164" Nov 02 21:45:39.130339 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.130304 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7"] Nov 02 21:45:39.133890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.133873 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.188275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.187967 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9cc56b\"" Nov 02 21:45:39.202873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.202834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.203055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.202888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxzj\" (UniqueName: \"kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj\") pod \"payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.237233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.237206 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9cc56b-dockercfg-tfw8g\"" Nov 02 21:45:39.239139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.239117 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7"] Nov 02 21:45:39.277715 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.277682 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc"] Nov 02 21:45:39.281107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.281087 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.303739 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.303710 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.303849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.303760 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxzj\" (UniqueName: \"kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj\") pod \"payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.304405 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.304388 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.339508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.339474 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc"] Nov 02 21:45:39.360142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.360117 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxzj\" (UniqueName: \"kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj\") pod \"payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.404616 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.404545 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wsk\" (UniqueName: \"kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk\") pod \"payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.404749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.404714 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.446453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.446433 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:39.505307 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.505253 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.505718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.505361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wsk\" (UniqueName: \"kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk\") pod \"payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.506633 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.506608 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.561275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.561248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wsk\" (UniqueName: \"kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk\") pod \"payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.594764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.594741 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:39.814849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.814823 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7"] Nov 02 21:45:39.816533 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:39.816495 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0ccbd5_e95f_43bb_8811_961a812f2bc7.slice/crio-c32531b127f069c8e541346fe44ff68d163c5f49f5c02d2d801d60b21bffe708 WatchSource:0}: Error finding container c32531b127f069c8e541346fe44ff68d163c5f49f5c02d2d801d60b21bffe708: Status 404 returned error can't find the container with id c32531b127f069c8e541346fe44ff68d163c5f49f5c02d2d801d60b21bffe708 Nov 02 21:45:39.964137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:39.964116 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc"] Nov 02 21:45:39.966719 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:39.966692 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35d741f_a717_424d_a22e_5fd1b834e96f.slice/crio-ba283de78fe26baea0ec0a2251c1cf661332c4202594cc37fe9cedd858c3560f WatchSource:0}: Error finding container ba283de78fe26baea0ec0a2251c1cf661332c4202594cc37fe9cedd858c3560f: Status 404 returned error can't find the container with id ba283de78fe26baea0ec0a2251c1cf661332c4202594cc37fe9cedd858c3560f Nov 02 21:45:40.479518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.479480 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" event={"ID":"c35d741f-a717-424d-a22e-5fd1b834e96f","Type":"ContainerStarted","Data":"09e33b3dd034c9b853d4937bc2f7825219af86a124871884b9d8749f70313728"} Nov 02 21:45:40.479767 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.479563 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" event={"ID":"c35d741f-a717-424d-a22e-5fd1b834e96f","Type":"ContainerStarted","Data":"ba283de78fe26baea0ec0a2251c1cf661332c4202594cc37fe9cedd858c3560f"} Nov 02 21:45:40.479767 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.479593 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:40.481111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.481086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" event={"ID":"5f0ccbd5-e95f-43bb-8811-961a812f2bc7","Type":"ContainerStarted","Data":"0e68cd9fb7faa7fac5fbe1740bc9834a236319ebad6bfecb0b8d8edc2fe5e23c"} Nov 02 21:45:40.481178 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.481133 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" event={"ID":"5f0ccbd5-e95f-43bb-8811-961a812f2bc7","Type":"ContainerStarted","Data":"c32531b127f069c8e541346fe44ff68d163c5f49f5c02d2d801d60b21bffe708"} Nov 02 21:45:40.481178 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.481159 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:40.593807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.593758 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" podStartSLOduration=1.593744783 podStartE2EDuration="1.593744783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:40.590933691 +0000 UTC m=+687.753028811" watchObservedRunningTime="2026-11-02 21:45:40.593744783 +0000 UTC m=+687.755839899" Nov 02 21:45:40.706705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:40.706638 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" podStartSLOduration=1.706625179 podStartE2EDuration="1.706625179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:40.705316627 +0000 UTC m=+687.867411746" watchObservedRunningTime="2026-11-02 21:45:40.706625179 +0000 UTC m=+687.868720290" Nov 02 21:45:46.403997 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.403956 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq"] Nov 02 21:45:46.407759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.407736 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.509727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.509692 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-10fdd5-dockercfg-jx8ls\"" Nov 02 21:45:46.511138 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.511115 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-10fdd5\"" Nov 02 21:45:46.512564 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.512537 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq"] Nov 02 21:45:46.561264 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.561233 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j"] Nov 02 21:45:46.564775 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.564759 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.567717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.567695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rhk\" (UniqueName: \"kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk\") pod \"payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.567812 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.567738 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.606984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.606889 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j"] Nov 02 21:45:46.668470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.668407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rhk\" (UniqueName: \"kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk\") pod \"payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.668470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.668455 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.668621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.668534 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.668621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.668561 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh2d\" (UniqueName: \"kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d\") pod \"payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.669253 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.669235 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.724601 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.724575 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rhk\" (UniqueName: \"kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk\") pod \"payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:46.769928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.769885 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.770105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.769934 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh2d\" (UniqueName: \"kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d\") pod \"payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.770606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.770585 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.826291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.826256 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh2d\" (UniqueName: \"kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d\") pod \"payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:46.878030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:46.878002 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:47.021129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.021097 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:47.257263 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.257189 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j"] Nov 02 21:45:47.260451 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:47.260425 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb295ea08_620f_4170_af89_fbc835dd9e15.slice/crio-de868b070aabf2da30d42c145664ebea8439afd6f6a1b1a9a7d95698755a298a WatchSource:0}: Error finding container de868b070aabf2da30d42c145664ebea8439afd6f6a1b1a9a7d95698755a298a: Status 404 returned error can't find the container with id de868b070aabf2da30d42c145664ebea8439afd6f6a1b1a9a7d95698755a298a Nov 02 21:45:47.393063 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.393040 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq"] Nov 02 21:45:47.394479 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:47.394453 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dec3ed9_b13f_4524_a76a_07227b75457a.slice/crio-9e77bd5a79c57df510bb9a35a00ca04c0b175696efd0a763e8fb33352f7d306e WatchSource:0}: Error finding container 9e77bd5a79c57df510bb9a35a00ca04c0b175696efd0a763e8fb33352f7d306e: Status 404 returned error can't find the container with id 9e77bd5a79c57df510bb9a35a00ca04c0b175696efd0a763e8fb33352f7d306e Nov 02 21:45:47.465118 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.465100 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 21:45:47.519908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.519839 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" event={"ID":"2dec3ed9-b13f-4524-a76a-07227b75457a","Type":"ContainerStarted","Data":"a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc"} Nov 02 21:45:47.519908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.519890 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" event={"ID":"2dec3ed9-b13f-4524-a76a-07227b75457a","Type":"ContainerStarted","Data":"9e77bd5a79c57df510bb9a35a00ca04c0b175696efd0a763e8fb33352f7d306e"} Nov 02 21:45:47.520049 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.519918 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:45:47.521362 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.521339 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" event={"ID":"b295ea08-620f-4170-af89-fbc835dd9e15","Type":"ContainerStarted","Data":"9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027"} Nov 02 21:45:47.521459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.521382 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" event={"ID":"b295ea08-620f-4170-af89-fbc835dd9e15","Type":"ContainerStarted","Data":"de868b070aabf2da30d42c145664ebea8439afd6f6a1b1a9a7d95698755a298a"} Nov 02 21:45:47.521503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.521468 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:47.695530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.695483 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" podStartSLOduration=1.695468582 podStartE2EDuration="1.695468582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:47.69353231 +0000 UTC m=+694.855627430" watchObservedRunningTime="2026-11-02 21:45:47.695468582 +0000 UTC m=+694.857563702" Nov 02 21:45:47.809588 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:47.809464 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" podStartSLOduration=1.809437695 podStartE2EDuration="1.809437695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:47.808193752 +0000 UTC m=+694.970288872" watchObservedRunningTime="2026-11-02 21:45:47.809437695 +0000 UTC m=+694.971532818" Nov 02 21:45:49.476709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:49.476656 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 21:45:51.168711 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.168657 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5"] Nov 02 21:45:51.172378 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.172356 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.228873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.228840 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5"] Nov 02 21:45:51.276106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.276066 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ffe386\"" Nov 02 21:45:51.276106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.276066 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ffe386-dockercfg-htwfc\"" Nov 02 21:45:51.313928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.313890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.314079 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.314029 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxk5x\" (UniqueName: \"kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x\") pod \"payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.327681 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.327642 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn"] Nov 02 21:45:51.331610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.331594 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.385043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.385010 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn"] Nov 02 21:45:51.415235 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.415175 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.415395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.415311 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.415395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.415365 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxk5x\" (UniqueName: \"kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x\") pod \"payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.415553 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.415478 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjk5\" (UniqueName: \"kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5\") pod \"payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.416027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.416009 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.471765 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.471689 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxk5x\" (UniqueName: \"kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x\") pod \"payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.487201 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.487174 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:51.489984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.489961 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 21:45:51.490091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.490031 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 21:45:51.515984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.515949 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.516133 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.516012 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjk5\" (UniqueName: \"kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5\") pod \"payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.516983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.516958 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.574183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.574146 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjk5\" (UniqueName: \"kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5\") pod \"payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.646279 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.646244 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:51.860248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:51.860223 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5"] Nov 02 21:45:51.861806 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:51.861777 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd54ecb3_9693_4cf3_b36c_4ce7483bf5cf.slice/crio-892d652dd7dd6670de1ef93ac886252518dc7c0041fa8c9b1b6250a048082862 WatchSource:0}: Error finding container 892d652dd7dd6670de1ef93ac886252518dc7c0041fa8c9b1b6250a048082862: Status 404 returned error can't find the container with id 892d652dd7dd6670de1ef93ac886252518dc7c0041fa8c9b1b6250a048082862 Nov 02 21:45:52.018504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.018481 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn"] Nov 02 21:45:52.019803 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:52.019775 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb8ba51_d617_41ba_8c4c_9d939c3ce13e.slice/crio-a38db4ecf78641460d05151e58b13520e5f5711ab70b143eef9d340230f33aa0 WatchSource:0}: Error finding container a38db4ecf78641460d05151e58b13520e5f5711ab70b143eef9d340230f33aa0: Status 404 returned error can't find the container with id a38db4ecf78641460d05151e58b13520e5f5711ab70b143eef9d340230f33aa0 Nov 02 21:45:52.550366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.550325 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" event={"ID":"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf","Type":"ContainerStarted","Data":"baf3ad22e0f711b04be34b0d490943a9b8a3f15e79f4bccd38bc8da02ce15de0"} Nov 02 21:45:52.550828 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.550383 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:45:52.550828 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.550404 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" event={"ID":"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf","Type":"ContainerStarted","Data":"892d652dd7dd6670de1ef93ac886252518dc7c0041fa8c9b1b6250a048082862"} Nov 02 21:45:52.551921 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.551896 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" event={"ID":"acb8ba51-d617-41ba-8c4c-9d939c3ce13e","Type":"ContainerStarted","Data":"7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825"} Nov 02 21:45:52.552014 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.551942 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:45:52.552014 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.551957 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" event={"ID":"acb8ba51-d617-41ba-8c4c-9d939c3ce13e","Type":"ContainerStarted","Data":"a38db4ecf78641460d05151e58b13520e5f5711ab70b143eef9d340230f33aa0"} Nov 02 21:45:52.665794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.665740 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" podStartSLOduration=1.6657257319999998 podStartE2EDuration="1.665725732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:52.663287743 +0000 UTC m=+699.825382866" watchObservedRunningTime="2026-11-02 21:45:52.665725732 +0000 UTC m=+699.827820852" Nov 02 21:45:52.779647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:52.779589 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" podStartSLOduration=1.7795740260000001 podStartE2EDuration="1.779574026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:52.777734546 +0000 UTC m=+699.939829667" watchObservedRunningTime="2026-11-02 21:45:52.779574026 +0000 UTC m=+699.941669176" Nov 02 21:45:53.615695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.615644 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6"] Nov 02 21:45:53.619505 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.619488 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.723340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.723299 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-96c052-dockercfg-4wz48\"" Nov 02 21:45:53.723513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.723449 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-96c052\"" Nov 02 21:45:53.726078 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.726052 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6"] Nov 02 21:45:53.738762 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.738734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx78\" (UniqueName: \"kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78\") pod \"payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.739040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.738993 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.779242 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.779218 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm"] Nov 02 21:45:53.782867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.782850 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:53.840699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.840647 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.840981 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.840829 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm"] Nov 02 21:45:53.841097 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.840948 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx78\" (UniqueName: \"kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78\") pod \"payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.841476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.841448 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.898877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.898850 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx78\" (UniqueName: \"kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78\") pod \"payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.933427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.933386 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:53.942531 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.942499 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:53.942649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:53.942546 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9nf\" (UniqueName: \"kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf\") pod \"payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:54.043603 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.043569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:54.043792 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.043618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9nf\" (UniqueName: \"kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf\") pod \"payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:54.044311 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.044285 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:54.100178 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.100148 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9nf\" (UniqueName: \"kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf\") pod \"payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:54.315472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.315381 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6"] Nov 02 21:45:54.328192 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:54.328159 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa92439_8d82_4647_8e08_4b9192d4bb9d.slice/crio-b192635e6fcf46bac08f37c590f9a1d5199c0efc88cbab597c1ee2e18f1753b0 WatchSource:0}: Error finding container b192635e6fcf46bac08f37c590f9a1d5199c0efc88cbab597c1ee2e18f1753b0: Status 404 returned error can't find the container with id b192635e6fcf46bac08f37c590f9a1d5199c0efc88cbab597c1ee2e18f1753b0 Nov 02 21:45:54.395417 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.395394 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:54.564089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.564045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" event={"ID":"eaa92439-8d82-4647-8e08-4b9192d4bb9d","Type":"ContainerStarted","Data":"070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29"} Nov 02 21:45:54.564244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.564102 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" event={"ID":"eaa92439-8d82-4647-8e08-4b9192d4bb9d","Type":"ContainerStarted","Data":"b192635e6fcf46bac08f37c590f9a1d5199c0efc88cbab597c1ee2e18f1753b0"} Nov 02 21:45:54.564244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.564139 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:45:54.677861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.677812 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" podStartSLOduration=1.677797189 podStartE2EDuration="1.677797189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:54.676832906 +0000 UTC m=+701.838928025" watchObservedRunningTime="2026-11-02 21:45:54.677797189 +0000 UTC m=+701.839892308" Nov 02 21:45:54.765075 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:54.765045 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm"] Nov 02 21:45:54.767649 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:45:54.767613 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c5a543_df95_42a4_8646_05e69bef7e43.slice/crio-73ff41eace52c07700213b359a5ca32aeb8ea0a0dc48a7e8b6e8867cb6c0979f WatchSource:0}: Error finding container 73ff41eace52c07700213b359a5ca32aeb8ea0a0dc48a7e8b6e8867cb6c0979f: Status 404 returned error can't find the container with id 73ff41eace52c07700213b359a5ca32aeb8ea0a0dc48a7e8b6e8867cb6c0979f Nov 02 21:45:55.572430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:55.572394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" event={"ID":"85c5a543-df95-42a4-8646-05e69bef7e43","Type":"ContainerStarted","Data":"38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832"} Nov 02 21:45:55.572613 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:55.572452 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" event={"ID":"85c5a543-df95-42a4-8646-05e69bef7e43","Type":"ContainerStarted","Data":"73ff41eace52c07700213b359a5ca32aeb8ea0a0dc48a7e8b6e8867cb6c0979f"} Nov 02 21:45:55.572702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:55.572631 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:45:55.686467 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:55.686422 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" podStartSLOduration=2.686407241 podStartE2EDuration="2.686407241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:55.684202022 +0000 UTC m=+702.846297144" watchObservedRunningTime="2026-11-02 21:45:55.686407241 +0000 UTC m=+702.848502360" Nov 02 21:45:58.528648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:58.528619 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 21:45:58.529124 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:45:58.528704 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 21:46:03.560050 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:03.560020 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 21:46:03.560504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:03.560094 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 21:46:05.575871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:05.575842 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 21:46:06.581207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:06.581172 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 21:46:07.112141 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:46:07.112095 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:46:21.136048 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.136010 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb"] Nov 02 21:46:21.143232 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.143211 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.196612 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.196582 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb"] Nov 02 21:46:21.208693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.208651 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.208816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.208707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.208816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.208787 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.208931 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.208912 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.208996 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.208979 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbb2j\" (UniqueName: \"kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.209055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.209035 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.243809 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.243780 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"simulator-w6-fb8112-kserve-self-signed-certs\"" Nov 02 21:46:21.243935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.243880 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"openshift-service-ca.crt\"" Nov 02 21:46:21.243935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.243885 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"default-dockercfg-h76tv\"" Nov 02 21:46:21.244903 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.244886 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"kube-root-ca.crt\"" Nov 02 21:46:21.309761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.309728 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.309959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.309796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbb2j\" (UniqueName: \"kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.309959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.309845 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.310089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.310007 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.310089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.310061 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.310196 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.310095 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.310529 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.310411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.310770 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.310725 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.312989 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.312961 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.313513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.313493 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.313865 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.313847 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.366230 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.366201 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbb2j\" (UniqueName: \"kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j\") pod \"simulator-w6-fb8112-kserve-665669945-dqsgb\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.455796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.455716 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:21.830311 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:21.830277 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb"] Nov 02 21:46:21.832514 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:46:21.832486 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod146b8ca0_a44d_4f48_a022_a4d335e92135.slice/crio-c67f1c3604b9e2f7f115db86bd7cbacb9c6d6fd66e79b14c58ea902f3231d793 WatchSource:0}: Error finding container c67f1c3604b9e2f7f115db86bd7cbacb9c6d6fd66e79b14c58ea902f3231d793: Status 404 returned error can't find the container with id c67f1c3604b9e2f7f115db86bd7cbacb9c6d6fd66e79b14c58ea902f3231d793 Nov 02 21:46:22.722865 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:22.722822 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" event={"ID":"146b8ca0-a44d-4f48-a022-a4d335e92135","Type":"ContainerStarted","Data":"c67f1c3604b9e2f7f115db86bd7cbacb9c6d6fd66e79b14c58ea902f3231d793"} Nov 02 21:46:24.735731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:24.735690 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" event={"ID":"146b8ca0-a44d-4f48-a022-a4d335e92135","Type":"ContainerStarted","Data":"dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1"} Nov 02 21:46:24.849526 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:24.849476 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podStartSLOduration=1.611633153 podStartE2EDuration="3.849462093s" totalImagesPullingTime="2.23782894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:24.846919292 +0000 UTC m=+732.009014414" watchObservedRunningTime="2026-11-02 21:46:24.849462093 +0000 UTC m=+732.011557210" Nov 02 21:46:25.456687 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:25.456639 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:31.456568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:31.456525 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:31.474924 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:31.474890 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:31.786356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:31.786276 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:46:37.113779 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:46:37.113743 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:46:46.944272 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:46.944242 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8"] Nov 02 21:46:46.947962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:46.947945 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.002469 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.002421 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"openshift-service-ca.crt\"" Nov 02 21:46:47.051816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.051788 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"kube-root-ca.crt\"" Nov 02 21:46:47.051953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.051814 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"default-dockercfg-2xl7c\"" Nov 02 21:46:47.052043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.052029 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"simulator-w2-9cc56b-kserve-self-signed-certs\"" Nov 02 21:46:47.053992 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.053973 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8"] Nov 02 21:46:47.057486 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.057469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.057544 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.057503 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.057654 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.057633 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.057776 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.057759 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.057824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.057807 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjz87\" (UniqueName: \"kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.058065 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.058035 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.158960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.158924 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.158989 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159019 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159127 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159371 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159177 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjz87\" (UniqueName: \"kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159401 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159446 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.159532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.159496 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.161404 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.161383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.161731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.161713 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.216017 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.215951 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjz87\" (UniqueName: \"kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87\") pod \"simulator-w2-9cc56b-kserve-68b98f446f-cf4j8\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.260303 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.259982 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:47.633876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.633846 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8"] Nov 02 21:46:47.635695 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:46:47.635647 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc9c8fbb_ef28_4236_a453_343ce87b22bc.slice/crio-50177b816a2aeabaa33c3cdbe13edeefd4ce3849a5b48da9c2e501e7c854fb01 WatchSource:0}: Error finding container 50177b816a2aeabaa33c3cdbe13edeefd4ce3849a5b48da9c2e501e7c854fb01: Status 404 returned error can't find the container with id 50177b816a2aeabaa33c3cdbe13edeefd4ce3849a5b48da9c2e501e7c854fb01 Nov 02 21:46:47.862943 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.862909 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" event={"ID":"dc9c8fbb-ef28-4236-a453-343ce87b22bc","Type":"ContainerStarted","Data":"c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca"} Nov 02 21:46:47.862943 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.862950 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" event={"ID":"dc9c8fbb-ef28-4236-a453-343ce87b22bc","Type":"ContainerStarted","Data":"50177b816a2aeabaa33c3cdbe13edeefd4ce3849a5b48da9c2e501e7c854fb01"} Nov 02 21:46:47.977072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:47.976980 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podStartSLOduration=1.9769668679999999 podStartE2EDuration="1.976966868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:47.975072817 +0000 UTC m=+755.137167928" watchObservedRunningTime="2026-11-02 21:46:47.976966868 +0000 UTC m=+755.139061989" Nov 02 21:46:48.260890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:48.260819 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:57.260900 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:57.260815 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:57.273499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:57.273473 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:46:57.936004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:46:57.935978 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:47:07.117154 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:47:07.117108 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:47:23.045801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.045769 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9"] Nov 02 21:47:23.050739 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.050723 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.105754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.105720 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.105913 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.105777 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.105913 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.105880 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.106045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.105904 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mfn\" (UniqueName: \"kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.106045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.105981 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.106045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.106018 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.131873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.131846 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"openshift-service-ca.crt\"" Nov 02 21:47:23.153451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.153420 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"kube-root-ca.crt\"" Nov 02 21:47:23.153743 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.153725 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"default-dockercfg-ls6zb\"" Nov 02 21:47:23.153831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.153791 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"simulator-w3-64438f-kserve-self-signed-certs\"" Nov 02 21:47:23.155938 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.155915 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9"] Nov 02 21:47:23.206560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.206526 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.206788 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.206572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mfn\" (UniqueName: \"kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.206788 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.206634 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.206788 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.206683 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.207059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.206799 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.207059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.206847 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.207059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.207038 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.207245 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.207224 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.207318 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.207298 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.209291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.209268 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.210859 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.210841 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.262700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.262649 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mfn\" (UniqueName: \"kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn\") pod \"simulator-w3-64438f-kserve-dbfc88845-6mtv9\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.363025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.362936 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:23.734690 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:23.734653 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9"] Nov 02 21:47:23.736517 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:47:23.736488 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb841d1a4_37c8_4403_8448_c61416c6341b.slice/crio-df7e279122b00848fa2d1e8cf5217e816916df3f7d7caacd6878a40f536daebe WatchSource:0}: Error finding container df7e279122b00848fa2d1e8cf5217e816916df3f7d7caacd6878a40f536daebe: Status 404 returned error can't find the container with id df7e279122b00848fa2d1e8cf5217e816916df3f7d7caacd6878a40f536daebe Nov 02 21:47:24.075441 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:24.075357 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" event={"ID":"b841d1a4-37c8-4403-8448-c61416c6341b","Type":"ContainerStarted","Data":"bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229"} Nov 02 21:47:24.075441 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:24.075402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" event={"ID":"b841d1a4-37c8-4403-8448-c61416c6341b","Type":"ContainerStarted","Data":"df7e279122b00848fa2d1e8cf5217e816916df3f7d7caacd6878a40f536daebe"} Nov 02 21:47:24.190639 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:24.190582 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podStartSLOduration=1.19056578 podStartE2EDuration="1.19056578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:24.188048734 +0000 UTC m=+791.350143859" watchObservedRunningTime="2026-11-02 21:47:24.19056578 +0000 UTC m=+791.352660898" Nov 02 21:47:24.364072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:24.363990 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:33.364124 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:33.364090 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:33.377450 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:33.377420 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:34.142653 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:34.142627 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:47:37.112718 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:47:37.112665 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:47:39.971006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:39.970971 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd"] Nov 02 21:47:39.974703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:39.974665 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.070935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.070898 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.071102 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.071044 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.071238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.071214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.071372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.071341 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.071434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.071419 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.071476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.071457 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngjc4\" (UniqueName: \"kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.078733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.078710 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ffe386\"/\"openshift-service-ca.crt\"" Nov 02 21:47:40.078923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.078898 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ffe386\"/\"simulator-w1-ffe386-kserve-self-signed-certs\"" Nov 02 21:47:40.078923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.078917 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ffe386\"/\"kube-root-ca.crt\"" Nov 02 21:47:40.079020 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.078932 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ffe386\"/\"default-dockercfg-rw9zm\"" Nov 02 21:47:40.081217 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.081196 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd"] Nov 02 21:47:40.172627 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.172596 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.172627 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.172635 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngjc4\" (UniqueName: \"kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.172917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.172664 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.172917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.172727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.172917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.172752 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.172917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.172830 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.173155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.173131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.173239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.173175 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.173239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.173227 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.175040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.175013 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.175319 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.175299 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.230971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.230903 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngjc4\" (UniqueName: \"kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4\") pod \"simulator-w1-ffe386-kserve-f9fb85db6-4mmvd\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.286635 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.286605 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:40.662011 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:40.661986 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd"] Nov 02 21:47:40.663405 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:47:40.663380 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a8e5011_11f6_4298_bc22_4f03b28ee856.slice/crio-0c5d7722ab209dd63ab358ba2ea0f24379455a36fb9dbd0168d607cd8b7f193c WatchSource:0}: Error finding container 0c5d7722ab209dd63ab358ba2ea0f24379455a36fb9dbd0168d607cd8b7f193c: Status 404 returned error can't find the container with id 0c5d7722ab209dd63ab358ba2ea0f24379455a36fb9dbd0168d607cd8b7f193c Nov 02 21:47:41.172741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:41.172705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" event={"ID":"4a8e5011-11f6-4298-bc22-4f03b28ee856","Type":"ContainerStarted","Data":"cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f"} Nov 02 21:47:41.173221 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:41.172758 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" event={"ID":"4a8e5011-11f6-4298-bc22-4f03b28ee856","Type":"ContainerStarted","Data":"0c5d7722ab209dd63ab358ba2ea0f24379455a36fb9dbd0168d607cd8b7f193c"} Nov 02 21:47:41.286872 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:41.286824 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podStartSLOduration=2.286811041 podStartE2EDuration="2.286811041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:41.284941354 +0000 UTC m=+808.447036473" watchObservedRunningTime="2026-11-02 21:47:41.286811041 +0000 UTC m=+808.448906160" Nov 02 21:47:41.287563 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:41.287540 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:50.287177 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:50.287142 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:50.299979 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:50.299951 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:47:51.241927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:47:51.241889 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:48:07.113597 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:48:07.113555 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:48:23.670948 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.670909 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75"] Nov 02 21:48:23.674868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.674849 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.727137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.727094 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb8112-openshift-default-dockercfg-r4kpc\"" Nov 02 21:48:23.779579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.779553 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75"] Nov 02 21:48:23.808434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808365 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808418 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808615 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808532 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808615 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808592 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808613 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtb9q\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808710 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808814 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808798 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.808878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.808860 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.909876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.909837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.909902 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.909963 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910000 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910034 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910267 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtb9q\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910407 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910317 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910407 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910356 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910407 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910365 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910407 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910391 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910608 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910407 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.910608 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.910548 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.911151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.911127 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.912362 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.912343 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.912518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.912498 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.968594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.968515 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:23.968766 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:23.968592 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtb9q\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:24.022107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:24.022078 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:24.443639 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:24.443616 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75"] Nov 02 21:48:24.445557 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:48:24.445527 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcdf4cc_c621_4f1e_9ade_144ecd5c3548.slice/crio-ae2b412a1d0fb2c0747167985864ae31cf1bc21e444bd6205ec343246ad4d48b WatchSource:0}: Error finding container ae2b412a1d0fb2c0747167985864ae31cf1bc21e444bd6205ec343246ad4d48b: Status 404 returned error can't find the container with id ae2b412a1d0fb2c0747167985864ae31cf1bc21e444bd6205ec343246ad4d48b Nov 02 21:48:24.448098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:24.448067 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:48:24.448182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:24.448127 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:48:24.448182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:24.448153 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:48:25.440649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:25.440612 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" event={"ID":"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548","Type":"ContainerStarted","Data":"1139b68c8dbf3474d516c880f16f29390573263fa86c4eee20a77f6d88b487c5"} Nov 02 21:48:25.441104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:25.440663 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" event={"ID":"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548","Type":"ContainerStarted","Data":"ae2b412a1d0fb2c0747167985864ae31cf1bc21e444bd6205ec343246ad4d48b"} Nov 02 21:48:25.605159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:25.605100 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podStartSLOduration=2.605086015 podStartE2EDuration="2.605086015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:25.603960126 +0000 UTC m=+852.766055246" watchObservedRunningTime="2026-11-02 21:48:25.605086015 +0000 UTC m=+852.767181133" Nov 02 21:48:26.022697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:26.022639 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:26.024508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:26.024481 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:48:26.024637 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:26.024538 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:48:27.022542 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:27.022511 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:48:27.022953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:27.022561 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:48:28.023019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:28.022979 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:48:28.023643 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:28.023058 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:48:29.023072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:29.023030 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:48:29.023462 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:29.023094 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:48:30.022910 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:30.022875 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:48:30.023119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:30.022927 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:48:31.037342 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:31.037288 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:31.037903 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:31.037877 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:31.038872 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:48:31.038852 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 21:48:37.112268 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:48:37.112230 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:49:07.114569 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:49:07.114520 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:49:13.092697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:13.092644 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:49:13.096103 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:13.096078 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:49:16.939463 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:16.939429 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs"] Nov 02 21:49:16.943219 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:16.943203 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.000957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.000927 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs"] Nov 02 21:49:17.047229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.047204 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ffe386\"/\"premium-w1-ffe386-kserve-self-signed-certs\"" Nov 02 21:49:17.062370 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.062340 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.062509 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.062378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.062573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.062522 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkxn\" (UniqueName: \"kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.062629 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.062583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.062724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.062694 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.062829 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.062809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.163857 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.163823 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkxn\" (UniqueName: \"kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.163876 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.163901 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.163954 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.164010 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.164050 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164406 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.164383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.164444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.164532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.164501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.166277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.166253 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.166611 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.166591 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.221007 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.220922 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkxn\" (UniqueName: \"kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn\") pod \"premium-w1-ffe386-kserve-6768f9f5c5-mckbs\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.256603 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.256561 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:17.655351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.655326 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs"] Nov 02 21:49:17.657336 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:49:17.657314 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod764023c0_d67a_451d_a0ec_56281fd4cfc3.slice/crio-e2055c5dd9a073c78f64fdd01a22a12bf2a568e533a006ad2ccbc27630e9092c WatchSource:0}: Error finding container e2055c5dd9a073c78f64fdd01a22a12bf2a568e533a006ad2ccbc27630e9092c: Status 404 returned error can't find the container with id e2055c5dd9a073c78f64fdd01a22a12bf2a568e533a006ad2ccbc27630e9092c Nov 02 21:49:17.757037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.757003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" event={"ID":"764023c0-d67a-451d-a0ec-56281fd4cfc3","Type":"ContainerStarted","Data":"c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956"} Nov 02 21:49:17.757200 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.757048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" event={"ID":"764023c0-d67a-451d-a0ec-56281fd4cfc3","Type":"ContainerStarted","Data":"e2055c5dd9a073c78f64fdd01a22a12bf2a568e533a006ad2ccbc27630e9092c"} Nov 02 21:49:17.869961 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:17.869900 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podStartSLOduration=1.869758142 podStartE2EDuration="1.869758142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:17.868912602 +0000 UTC m=+905.031007723" watchObservedRunningTime="2026-11-02 21:49:17.869758142 +0000 UTC m=+905.031853263" Nov 02 21:49:18.256889 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:18.256851 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:23.024109 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.024074 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4"] Nov 02 21:49:23.028295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.028272 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.083478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.083450 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4"] Nov 02 21:49:23.119860 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.119828 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.120046 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.120022 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqsbt\" (UniqueName: \"kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.120091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.120058 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.129734 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.129706 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:49:23.129876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.129818 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:49:23.129967 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.129905 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:49:23.129967 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.129937 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:49:23.221389 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.221351 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqsbt\" (UniqueName: \"kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.221530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.221398 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.221530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.221445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.222182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.222160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.223666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.223649 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.278394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.278333 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqsbt\" (UniqueName: \"kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt\") pod \"maas-api-revoke-sub-2dd1404c-m8pc4\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.341808 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.341780 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:23.717042 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.717014 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4"] Nov 02 21:49:23.718715 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:49:23.718664 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff05d887_ddf1_4d71_a63c_326cabd4b1d6.slice/crio-a81c8b90e84a3017bcd1d3b110004ab7fbf48fbdd3602cf0bab16f67443114e1 WatchSource:0}: Error finding container a81c8b90e84a3017bcd1d3b110004ab7fbf48fbdd3602cf0bab16f67443114e1: Status 404 returned error can't find the container with id a81c8b90e84a3017bcd1d3b110004ab7fbf48fbdd3602cf0bab16f67443114e1 Nov 02 21:49:23.720731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.720713 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:49:23.802736 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:23.802696 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" event={"ID":"ff05d887-ddf1-4d71-a63c-326cabd4b1d6","Type":"ContainerStarted","Data":"a81c8b90e84a3017bcd1d3b110004ab7fbf48fbdd3602cf0bab16f67443114e1"} Nov 02 21:49:27.256861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:27.256825 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:27.270874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:27.270841 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:27.840465 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:27.840435 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:49:32.858393 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:32.858361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" event={"ID":"ff05d887-ddf1-4d71-a63c-326cabd4b1d6","Type":"ContainerStarted","Data":"df73fd6ccada05ca6b1ec717a79793c0b783bc33f76ec0e3771f504363f721b9"} Nov 02 21:49:32.971725 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:32.971649 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" podStartSLOduration=2.2594977800000002 podStartE2EDuration="10.971633132s" totalImagesPullingTime="8.712135352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:32.97033093 +0000 UTC m=+920.132426051" watchObservedRunningTime="2026-11-02 21:49:32.971633132 +0000 UTC m=+920.133728252" Nov 02 21:49:33.865383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:33.865345 5091 generic.go:356] "Generic (PLEG): container finished" podID="ff05d887-ddf1-4d71-a63c-326cabd4b1d6" containerID="df73fd6ccada05ca6b1ec717a79793c0b783bc33f76ec0e3771f504363f721b9" exitCode=0 Nov 02 21:49:33.865791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:33.865425 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" event={"ID":"ff05d887-ddf1-4d71-a63c-326cabd4b1d6","Type":"ContainerDied","Data":"df73fd6ccada05ca6b1ec717a79793c0b783bc33f76ec0e3771f504363f721b9"} Nov 02 21:49:35.016106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.016080 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:35.041318 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.041293 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt\" (UniqueName: \"kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt\") pod \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " Nov 02 21:49:35.041448 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.041385 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca\") pod \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " Nov 02 21:49:35.041531 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.041514 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca\") pod \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\" (UID: \"ff05d887-ddf1-4d71-a63c-326cabd4b1d6\") " Nov 02 21:49:35.041747 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.041721 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca" pod "ff05d887-ddf1-4d71-a63c-326cabd4b1d6" (UID: "ff05d887-ddf1-4d71-a63c-326cabd4b1d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:35.042099 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.042072 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:49:35.043525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.043507 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt" pod "ff05d887-ddf1-4d71-a63c-326cabd4b1d6" (UID: "ff05d887-ddf1-4d71-a63c-326cabd4b1d6"). InnerVolumeSpecName "kube-api-access-bqsbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:35.043612 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.043593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca" pod "ff05d887-ddf1-4d71-a63c-326cabd4b1d6" (UID: "ff05d887-ddf1-4d71-a63c-326cabd4b1d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:35.144962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.144890 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:49:35.144962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.144921 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqsbt\" (UniqueName: \"kubernetes.io/projected/ff05d887-ddf1-4d71-a63c-326cabd4b1d6-kube-api-access-bqsbt\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:49:35.878457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.878428 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" Nov 02 21:49:35.878655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.878432 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4" event={"ID":"ff05d887-ddf1-4d71-a63c-326cabd4b1d6","Type":"ContainerDied","Data":"a81c8b90e84a3017bcd1d3b110004ab7fbf48fbdd3602cf0bab16f67443114e1"} Nov 02 21:49:35.878655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:35.878559 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a81c8b90e84a3017bcd1d3b110004ab7fbf48fbdd3602cf0bab16f67443114e1" Nov 02 21:49:37.112575 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:49:37.112540 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:49:40.491425 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.491375 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2"] Nov 02 21:49:40.492220 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.492196 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ff05d887-ddf1-4d71-a63c-326cabd4b1d6" containerName="revoke-keys" Nov 02 21:49:40.492322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.492222 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff05d887-ddf1-4d71-a63c-326cabd4b1d6" containerName="revoke-keys" Nov 02 21:49:40.492418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.492404 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff05d887-ddf1-4d71-a63c-326cabd4b1d6" containerName="revoke-keys" Nov 02 21:49:40.496504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.496476 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.548387 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.548345 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-10fdd5-openshift-default-dockercfg-pg4pr\"" Nov 02 21:49:40.601458 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601412 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.601645 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601498 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.601725 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601652 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.601794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601765 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmh2\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.601880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601860 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.601936 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601916 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.601983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601953 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.602032 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.601999 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.602112 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.602094 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.603067 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.603042 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2"] Nov 02 21:49:40.705272 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705514 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705616 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705646 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705692 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705728 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705769 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmh2\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.705912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.705886 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.706797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.706770 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.707087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.707062 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.707737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.707714 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.709344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.708352 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.709483 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.708879 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.729737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.723945 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.729737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.724212 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.766153 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.765774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.767476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.767446 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmh2\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:40.817586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:40.817554 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:41.252470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:41.252444 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2"] Nov 02 21:49:41.254635 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:49:41.254605 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f7f515_7919_4c2c_bb66_42a3054bc682.slice/crio-11399d8d89493c01cec4858db3175889e43baebfcb50f25cc06814a6151e56d0 WatchSource:0}: Error finding container 11399d8d89493c01cec4858db3175889e43baebfcb50f25cc06814a6151e56d0: Status 404 returned error can't find the container with id 11399d8d89493c01cec4858db3175889e43baebfcb50f25cc06814a6151e56d0 Nov 02 21:49:41.257045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:41.257013 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:41.257137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:41.257079 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:41.257137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:41.257108 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:41.916030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:41.915994 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" event={"ID":"76f7f515-7919-4c2c-bb66-42a3054bc682","Type":"ContainerStarted","Data":"25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c"} Nov 02 21:49:41.916454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:41.916053 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" event={"ID":"76f7f515-7919-4c2c-bb66-42a3054bc682","Type":"ContainerStarted","Data":"11399d8d89493c01cec4858db3175889e43baebfcb50f25cc06814a6151e56d0"} Nov 02 21:49:42.051483 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:42.051429 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podStartSLOduration=2.051414317 podStartE2EDuration="2.051414317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:42.04954242 +0000 UTC m=+929.211637542" watchObservedRunningTime="2026-11-02 21:49:42.051414317 +0000 UTC m=+929.213509515" Nov 02 21:49:42.817922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:42.817889 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:42.819652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:42.819625 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:49:42.819794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:42.819700 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:49:43.818853 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:43.818814 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:49:43.819333 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:43.818870 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:49:44.820958 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:44.818747 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:49:44.820958 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:44.818813 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:49:45.818344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:45.818311 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:49:45.818525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:45.818368 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:49:46.818874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:46.818841 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:49:46.819291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:46.818897 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:49:47.910093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:47.910047 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:47.910797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:47.910770 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:47.911521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:47.911498 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 21:49:48.502523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.502491 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq"] Nov 02 21:49:48.507416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.507392 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.561741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.561711 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:49:48.561741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.561721 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:49:48.561969 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.561785 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:49:48.561969 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.561801 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:49:48.565399 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.565380 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq"] Nov 02 21:49:48.625015 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.624989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.625149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.625018 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpbw\" (UniqueName: \"kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.625149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.625117 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.725877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.725838 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.726038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.725879 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpbw\" (UniqueName: \"kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.726038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.725921 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.726658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.726625 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.728228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.728206 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.782375 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.782308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpbw\" (UniqueName: \"kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw\") pod \"maas-api-revoke-sub-1b806723-s8ngq\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:48.823713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:48.823688 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:49.197142 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:49.197118 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq"] Nov 02 21:49:49.198934 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:49:49.198909 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a2aea8_632e_4092_ad4f_b0949d66a3ec.slice/crio-fe542a533e5655d744adabdb99dcc0a3ade3f487dd33bccd968d512158aadb2f WatchSource:0}: Error finding container fe542a533e5655d744adabdb99dcc0a3ade3f487dd33bccd968d512158aadb2f: Status 404 returned error can't find the container with id fe542a533e5655d744adabdb99dcc0a3ade3f487dd33bccd968d512158aadb2f Nov 02 21:49:49.980393 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:49.980360 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" event={"ID":"d4a2aea8-632e-4092-ad4f-b0949d66a3ec","Type":"ContainerStarted","Data":"6db9b86202db07b7871c16c503e9f6230fafc57ce554c9e441f54364d29001f6"} Nov 02 21:49:49.980586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:49.980410 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" event={"ID":"d4a2aea8-632e-4092-ad4f-b0949d66a3ec","Type":"ContainerStarted","Data":"fe542a533e5655d744adabdb99dcc0a3ade3f487dd33bccd968d512158aadb2f"} Nov 02 21:49:50.094421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:50.094377 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" podStartSLOduration=2.094361706 podStartE2EDuration="2.094361706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:50.091974033 +0000 UTC m=+937.254069152" watchObservedRunningTime="2026-11-02 21:49:50.094361706 +0000 UTC m=+937.256456827" Nov 02 21:49:50.986807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:50.986763 5091 generic.go:356] "Generic (PLEG): container finished" podID="d4a2aea8-632e-4092-ad4f-b0949d66a3ec" containerID="6db9b86202db07b7871c16c503e9f6230fafc57ce554c9e441f54364d29001f6" exitCode=0 Nov 02 21:49:50.987186 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:50.986850 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" event={"ID":"d4a2aea8-632e-4092-ad4f-b0949d66a3ec","Type":"ContainerDied","Data":"6db9b86202db07b7871c16c503e9f6230fafc57ce554c9e441f54364d29001f6"} Nov 02 21:49:52.138565 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.138541 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:52.263127 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.263054 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca\") pod \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " Nov 02 21:49:52.263281 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.263132 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw\" (UniqueName: \"kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw\") pod \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " Nov 02 21:49:52.263344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.263276 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca\") pod \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\" (UID: \"d4a2aea8-632e-4092-ad4f-b0949d66a3ec\") " Nov 02 21:49:52.263475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.263446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca" pod "d4a2aea8-632e-4092-ad4f-b0949d66a3ec" (UID: "d4a2aea8-632e-4092-ad4f-b0949d66a3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:52.263619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.263601 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:49:52.265361 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.265332 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca" pod "d4a2aea8-632e-4092-ad4f-b0949d66a3ec" (UID: "d4a2aea8-632e-4092-ad4f-b0949d66a3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:52.265494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.265472 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw" pod "d4a2aea8-632e-4092-ad4f-b0949d66a3ec" (UID: "d4a2aea8-632e-4092-ad4f-b0949d66a3ec"). InnerVolumeSpecName "kube-api-access-fgpbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:52.364459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.364428 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:49:52.364459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:52.364457 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgpbw\" (UniqueName: \"kubernetes.io/projected/d4a2aea8-632e-4092-ad4f-b0949d66a3ec-kube-api-access-fgpbw\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:49:53.000423 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:53.000397 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" Nov 02 21:49:53.000556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:53.000438 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq" event={"ID":"d4a2aea8-632e-4092-ad4f-b0949d66a3ec","Type":"ContainerDied","Data":"fe542a533e5655d744adabdb99dcc0a3ade3f487dd33bccd968d512158aadb2f"} Nov 02 21:49:53.000556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:49:53.000480 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe542a533e5655d744adabdb99dcc0a3ade3f487dd33bccd968d512158aadb2f" Nov 02 21:50:02.457013 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.456931 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5"] Nov 02 21:50:02.457445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.457431 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d4a2aea8-632e-4092-ad4f-b0949d66a3ec" containerName="revoke-keys" Nov 02 21:50:02.457486 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.457446 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a2aea8-632e-4092-ad4f-b0949d66a3ec" containerName="revoke-keys" Nov 02 21:50:02.457537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.457529 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a2aea8-632e-4092-ad4f-b0949d66a3ec" containerName="revoke-keys" Nov 02 21:50:02.460816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.460796 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.564413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.564383 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:02.564413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.564389 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:02.564652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.564391 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:50:02.564652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.564449 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:02.567044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.567023 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5"] Nov 02 21:50:02.568939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.568920 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.569004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.568947 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.569085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.569065 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmqb\" (UniqueName: \"kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.669991 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.669962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmqb\" (UniqueName: \"kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.670126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.670037 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.670126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.670053 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.670721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.670696 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.672326 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.672305 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.726170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.726115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmqb\" (UniqueName: \"kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb\") pod \"maas-api-revoke-sub-fd65be97-nmsr5\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:02.774226 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:02.774203 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:03.143418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:03.143393 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5"] Nov 02 21:50:03.144606 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:03.144584 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239deac6_670a_4b0a_a61b_f34af8bc8b15.slice/crio-13ff3dca1a57cc3d6b5f4421336243e890d02f1c58d9b071c862c84c20bdd5d0 WatchSource:0}: Error finding container 13ff3dca1a57cc3d6b5f4421336243e890d02f1c58d9b071c862c84c20bdd5d0: Status 404 returned error can't find the container with id 13ff3dca1a57cc3d6b5f4421336243e890d02f1c58d9b071c862c84c20bdd5d0 Nov 02 21:50:03.987426 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:50:03.987349 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239deac6_670a_4b0a_a61b_f34af8bc8b15.slice/crio-a0b3a648d525932e439d900358633af3636f1d3a73258db84640fd8d5fe5dc5a.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:50:03.993751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:03.992807 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2"] Nov 02 21:50:03.997513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:03.997492 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.050327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.050292 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2"] Nov 02 21:50:04.064150 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.064121 5091 generic.go:356] "Generic (PLEG): container finished" podID="239deac6-670a-4b0a-a61b-f34af8bc8b15" containerID="a0b3a648d525932e439d900358633af3636f1d3a73258db84640fd8d5fe5dc5a" exitCode=0 Nov 02 21:50:04.064294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.064208 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" event={"ID":"239deac6-670a-4b0a-a61b-f34af8bc8b15","Type":"ContainerDied","Data":"a0b3a648d525932e439d900358633af3636f1d3a73258db84640fd8d5fe5dc5a"} Nov 02 21:50:04.064294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.064263 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" event={"ID":"239deac6-670a-4b0a-a61b-f34af8bc8b15","Type":"ContainerStarted","Data":"13ff3dca1a57cc3d6b5f4421336243e890d02f1c58d9b071c862c84c20bdd5d0"} Nov 02 21:50:04.083154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.083121 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.083275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.083244 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2r4c\" (UniqueName: \"kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.083344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.083279 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.184257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.184229 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.184400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.184333 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2r4c\" (UniqueName: \"kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.184400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.184381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.185128 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.185097 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.186773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.186749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.242299 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.242230 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2r4c\" (UniqueName: \"kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c\") pod \"maas-api-revoke-sub-e59308df-bpdd2\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.340520 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.340485 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:04.716885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:04.716857 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2"] Nov 02 21:50:04.718236 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:04.718215 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda448efd3_f9a1_435d_b777_70922943cf73.slice/crio-245cc2e9a6ce67eba2b8a936ec2a8140f76be60d0b3d8824907219973cc202ce WatchSource:0}: Error finding container 245cc2e9a6ce67eba2b8a936ec2a8140f76be60d0b3d8824907219973cc202ce: Status 404 returned error can't find the container with id 245cc2e9a6ce67eba2b8a936ec2a8140f76be60d0b3d8824907219973cc202ce Nov 02 21:50:05.071560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.071526 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" event={"ID":"a448efd3-f9a1-435d-b777-70922943cf73","Type":"ContainerStarted","Data":"245cc2e9a6ce67eba2b8a936ec2a8140f76be60d0b3d8824907219973cc202ce"} Nov 02 21:50:05.210656 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.210626 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:05.294562 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.294489 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca\") pod \"239deac6-670a-4b0a-a61b-f34af8bc8b15\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " Nov 02 21:50:05.294562 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.294524 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca\") pod \"239deac6-670a-4b0a-a61b-f34af8bc8b15\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " Nov 02 21:50:05.294748 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.294566 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb\" (UniqueName: \"kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb\") pod \"239deac6-670a-4b0a-a61b-f34af8bc8b15\" (UID: \"239deac6-670a-4b0a-a61b-f34af8bc8b15\") " Nov 02 21:50:05.294951 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.294921 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca" pod "239deac6-670a-4b0a-a61b-f34af8bc8b15" (UID: "239deac6-670a-4b0a-a61b-f34af8bc8b15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:05.296593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.296571 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca" pod "239deac6-670a-4b0a-a61b-f34af8bc8b15" (UID: "239deac6-670a-4b0a-a61b-f34af8bc8b15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:05.296681 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.296630 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb" pod "239deac6-670a-4b0a-a61b-f34af8bc8b15" (UID: "239deac6-670a-4b0a-a61b-f34af8bc8b15"). InnerVolumeSpecName "kube-api-access-lwmqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:05.396117 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.396079 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:05.396117 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.396110 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/239deac6-670a-4b0a-a61b-f34af8bc8b15-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:05.396297 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:05.396126 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmqb\" (UniqueName: \"kubernetes.io/projected/239deac6-670a-4b0a-a61b-f34af8bc8b15-kube-api-access-lwmqb\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.079935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:06.079853 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" Nov 02 21:50:06.079935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:06.079882 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5" event={"ID":"239deac6-670a-4b0a-a61b-f34af8bc8b15","Type":"ContainerDied","Data":"13ff3dca1a57cc3d6b5f4421336243e890d02f1c58d9b071c862c84c20bdd5d0"} Nov 02 21:50:06.079935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:06.079936 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ff3dca1a57cc3d6b5f4421336243e890d02f1c58d9b071c862c84c20bdd5d0" Nov 02 21:50:06.081798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:06.081772 5091 generic.go:356] "Generic (PLEG): container finished" podID="a448efd3-f9a1-435d-b777-70922943cf73" containerID="804061ab55f3b31cd1271096f2d35f18a00b48b0bd59052b7a30a42b11363385" exitCode=0 Nov 02 21:50:06.081912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:06.081864 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" event={"ID":"a448efd3-f9a1-435d-b777-70922943cf73","Type":"ContainerDied","Data":"804061ab55f3b31cd1271096f2d35f18a00b48b0bd59052b7a30a42b11363385"} Nov 02 21:50:07.148498 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:50:07.147861 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:50:07.387652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.387625 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:07.428660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.428623 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca\") pod \"a448efd3-f9a1-435d-b777-70922943cf73\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " Nov 02 21:50:07.428834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.428692 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c\" (UniqueName: \"kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c\") pod \"a448efd3-f9a1-435d-b777-70922943cf73\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " Nov 02 21:50:07.428834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.428791 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca\") pod \"a448efd3-f9a1-435d-b777-70922943cf73\" (UID: \"a448efd3-f9a1-435d-b777-70922943cf73\") " Nov 02 21:50:07.429151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.429128 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca" pod "a448efd3-f9a1-435d-b777-70922943cf73" (UID: "a448efd3-f9a1-435d-b777-70922943cf73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:07.430736 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.430711 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca" pod "a448efd3-f9a1-435d-b777-70922943cf73" (UID: "a448efd3-f9a1-435d-b777-70922943cf73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:07.430861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.430824 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c" pod "a448efd3-f9a1-435d-b777-70922943cf73" (UID: "a448efd3-f9a1-435d-b777-70922943cf73"). InnerVolumeSpecName "kube-api-access-b2r4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:07.530354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.530323 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:07.530354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.530355 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2r4c\" (UniqueName: \"kubernetes.io/projected/a448efd3-f9a1-435d-b777-70922943cf73-kube-api-access-b2r4c\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:07.530560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:07.530371 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a448efd3-f9a1-435d-b777-70922943cf73-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:08.102258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:08.102223 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" Nov 02 21:50:08.102429 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:08.102227 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2" event={"ID":"a448efd3-f9a1-435d-b777-70922943cf73","Type":"ContainerDied","Data":"245cc2e9a6ce67eba2b8a936ec2a8140f76be60d0b3d8824907219973cc202ce"} Nov 02 21:50:08.102429 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:08.102357 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245cc2e9a6ce67eba2b8a936ec2a8140f76be60d0b3d8824907219973cc202ce" Nov 02 21:50:13.240776 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.240741 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl"] Nov 02 21:50:13.241251 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.241236 5091 cpu_manager.go:414] "containerMap: removing container" podUID="239deac6-670a-4b0a-a61b-f34af8bc8b15" containerName="revoke-keys" Nov 02 21:50:13.241298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.241252 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239deac6-670a-4b0a-a61b-f34af8bc8b15" containerName="revoke-keys" Nov 02 21:50:13.241298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.241281 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a448efd3-f9a1-435d-b777-70922943cf73" containerName="revoke-keys" Nov 02 21:50:13.241298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.241286 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a448efd3-f9a1-435d-b777-70922943cf73" containerName="revoke-keys" Nov 02 21:50:13.241391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.241368 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a448efd3-f9a1-435d-b777-70922943cf73" containerName="revoke-keys" Nov 02 21:50:13.241391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.241380 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="239deac6-670a-4b0a-a61b-f34af8bc8b15" containerName="revoke-keys" Nov 02 21:50:13.244724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.244707 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.295925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.295898 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.296052 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.295963 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnb4g\" (UniqueName: \"kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.296104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.296085 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.302366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.302342 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl"] Nov 02 21:50:13.348584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.348553 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:13.348803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.348597 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:13.348803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.348600 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:13.348803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.348660 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:50:13.397324 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.397300 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.397478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.397350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnb4g\" (UniqueName: \"kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.397478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.397410 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.398027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.397978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.399721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.399698 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.453550 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.453521 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnb4g\" (UniqueName: \"kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g\") pod \"maas-api-revoke-sub-866b5736-xh5pl\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.557896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.557814 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:13.928729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:13.928701 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl"] Nov 02 21:50:13.929478 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:13.929456 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf946dfc7_847b_40c3_86ee_fa5fddc5bd35.slice/crio-e11ca9ebaf2717c4e4a7e87fabaa606204588d509ad02c0c59de0973de0cf579 WatchSource:0}: Error finding container e11ca9ebaf2717c4e4a7e87fabaa606204588d509ad02c0c59de0973de0cf579: Status 404 returned error can't find the container with id e11ca9ebaf2717c4e4a7e87fabaa606204588d509ad02c0c59de0973de0cf579 Nov 02 21:50:14.147504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:14.147468 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" event={"ID":"f946dfc7-847b-40c3-86ee-fa5fddc5bd35","Type":"ContainerStarted","Data":"e11ca9ebaf2717c4e4a7e87fabaa606204588d509ad02c0c59de0973de0cf579"} Nov 02 21:50:15.154879 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:15.154841 5091 generic.go:356] "Generic (PLEG): container finished" podID="f946dfc7-847b-40c3-86ee-fa5fddc5bd35" containerID="9d7d42d8307aef00b574121e2b4476a5592e5b6fea7c6abe3db5d8dce9f799d9" exitCode=0 Nov 02 21:50:15.155315 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:15.154927 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" event={"ID":"f946dfc7-847b-40c3-86ee-fa5fddc5bd35","Type":"ContainerDied","Data":"9d7d42d8307aef00b574121e2b4476a5592e5b6fea7c6abe3db5d8dce9f799d9"} Nov 02 21:50:16.312346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.312309 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:16.426804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.426723 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g\" (UniqueName: \"kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g\") pod \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " Nov 02 21:50:16.427037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.426807 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca\") pod \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " Nov 02 21:50:16.427037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.426901 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca\") pod \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\" (UID: \"f946dfc7-847b-40c3-86ee-fa5fddc5bd35\") " Nov 02 21:50:16.427591 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.427562 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca" pod "f946dfc7-847b-40c3-86ee-fa5fddc5bd35" (UID: "f946dfc7-847b-40c3-86ee-fa5fddc5bd35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:16.429586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.429556 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca" pod "f946dfc7-847b-40c3-86ee-fa5fddc5bd35" (UID: "f946dfc7-847b-40c3-86ee-fa5fddc5bd35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:16.429718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.429556 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g" pod "f946dfc7-847b-40c3-86ee-fa5fddc5bd35" (UID: "f946dfc7-847b-40c3-86ee-fa5fddc5bd35"). InnerVolumeSpecName "kube-api-access-vnb4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:16.528508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.528477 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:16.528508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.528506 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnb4g\" (UniqueName: \"kubernetes.io/projected/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-kube-api-access-vnb4g\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:16.528508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:16.528516 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f946dfc7-847b-40c3-86ee-fa5fddc5bd35-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:17.168520 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:17.168487 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" event={"ID":"f946dfc7-847b-40c3-86ee-fa5fddc5bd35","Type":"ContainerDied","Data":"e11ca9ebaf2717c4e4a7e87fabaa606204588d509ad02c0c59de0973de0cf579"} Nov 02 21:50:17.168690 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:17.168533 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e11ca9ebaf2717c4e4a7e87fabaa606204588d509ad02c0c59de0973de0cf579" Nov 02 21:50:17.168690 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:17.168498 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl" Nov 02 21:50:29.060532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.060501 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s"] Nov 02 21:50:29.061002 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.060988 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f946dfc7-847b-40c3-86ee-fa5fddc5bd35" containerName="revoke-keys" Nov 02 21:50:29.061049 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.061002 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f946dfc7-847b-40c3-86ee-fa5fddc5bd35" containerName="revoke-keys" Nov 02 21:50:29.061111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.061102 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f946dfc7-847b-40c3-86ee-fa5fddc5bd35" containerName="revoke-keys" Nov 02 21:50:29.064460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.064438 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.121132 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.121101 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s"] Nov 02 21:50:29.167343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.167312 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:29.167552 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.167384 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:29.167552 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.167417 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:50:29.167724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.167426 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:29.177177 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.177151 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.177280 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.177227 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8flx\" (UniqueName: \"kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.177368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.177345 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.278869 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.278836 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.279033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.278894 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8flx\" (UniqueName: \"kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.279033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.278935 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.279699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.279659 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.281365 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.281334 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.335380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.335326 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8flx\" (UniqueName: \"kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx\") pod \"maas-api-revoke-sub-453ce505-qr87s\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.377813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.377789 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:29.749820 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:29.749785 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s"] Nov 02 21:50:29.751224 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:29.751178 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef03a9e9_aff6_4c95_ab84_882758a06b61.slice/crio-9469d550030978d65739fea8c63bac4598603d265dba88cc02f4332a93fdd4fe WatchSource:0}: Error finding container 9469d550030978d65739fea8c63bac4598603d265dba88cc02f4332a93fdd4fe: Status 404 returned error can't find the container with id 9469d550030978d65739fea8c63bac4598603d265dba88cc02f4332a93fdd4fe Nov 02 21:50:30.248272 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:30.248239 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" event={"ID":"ef03a9e9-aff6-4c95-ab84-882758a06b61","Type":"ContainerStarted","Data":"e6ac5e89cd835aa3ce546fade4c9a7cd4f470d3dcfaf9facbd7fe0e2b8dff5fb"} Nov 02 21:50:30.248637 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:30.248287 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" event={"ID":"ef03a9e9-aff6-4c95-ab84-882758a06b61","Type":"ContainerStarted","Data":"9469d550030978d65739fea8c63bac4598603d265dba88cc02f4332a93fdd4fe"} Nov 02 21:50:30.362712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:30.362600 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" podStartSLOduration=1.3625854099999999 podStartE2EDuration="1.36258541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:30.360898633 +0000 UTC m=+977.522993753" watchObservedRunningTime="2026-11-02 21:50:30.36258541 +0000 UTC m=+977.524680539" Nov 02 21:50:31.256061 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.256029 5091 generic.go:356] "Generic (PLEG): container finished" podID="ef03a9e9-aff6-4c95-ab84-882758a06b61" containerID="e6ac5e89cd835aa3ce546fade4c9a7cd4f470d3dcfaf9facbd7fe0e2b8dff5fb" exitCode=0 Nov 02 21:50:31.256476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.256106 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" event={"ID":"ef03a9e9-aff6-4c95-ab84-882758a06b61","Type":"ContainerDied","Data":"e6ac5e89cd835aa3ce546fade4c9a7cd4f470d3dcfaf9facbd7fe0e2b8dff5fb"} Nov 02 21:50:31.648538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.648501 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5"] Nov 02 21:50:31.653727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.653703 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.708129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.708100 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5"] Nov 02 21:50:31.808572 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.807203 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.808572 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.807309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrb9\" (UniqueName: \"kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.808572 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.807432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.908867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.908787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.908867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.908856 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.909080 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.908931 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrb9\" (UniqueName: \"kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.909533 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.909504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.911180 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.911157 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.965856 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.965824 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrb9\" (UniqueName: \"kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9\") pod \"maas-api-revoke-sub-5979a010-dh7q5\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:31.967620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:31.967595 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:32.340054 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:32.340020 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c182a9_2d44_4204_a23e_064e717739bd.slice/crio-83c7d3e1a24f14ad6fee1525356016159a2c3ac501363aca80791ba6fa477601 WatchSource:0}: Error finding container 83c7d3e1a24f14ad6fee1525356016159a2c3ac501363aca80791ba6fa477601: Status 404 returned error can't find the container with id 83c7d3e1a24f14ad6fee1525356016159a2c3ac501363aca80791ba6fa477601 Nov 02 21:50:32.340478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.340224 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5"] Nov 02 21:50:32.397998 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.397973 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:32.516107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.516024 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca\") pod \"ef03a9e9-aff6-4c95-ab84-882758a06b61\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " Nov 02 21:50:32.516107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.516094 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx\" (UniqueName: \"kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx\") pod \"ef03a9e9-aff6-4c95-ab84-882758a06b61\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " Nov 02 21:50:32.516326 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.516150 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca\") pod \"ef03a9e9-aff6-4c95-ab84-882758a06b61\" (UID: \"ef03a9e9-aff6-4c95-ab84-882758a06b61\") " Nov 02 21:50:32.516435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.516410 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca" pod "ef03a9e9-aff6-4c95-ab84-882758a06b61" (UID: "ef03a9e9-aff6-4c95-ab84-882758a06b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:32.516606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.516586 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.518780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.518752 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx" pod "ef03a9e9-aff6-4c95-ab84-882758a06b61" (UID: "ef03a9e9-aff6-4c95-ab84-882758a06b61"). InnerVolumeSpecName "kube-api-access-w8flx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:32.518892 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.518843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca" pod "ef03a9e9-aff6-4c95-ab84-882758a06b61" (UID: "ef03a9e9-aff6-4c95-ab84-882758a06b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:32.572981 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.572948 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj"] Nov 02 21:50:32.580719 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.574028 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ef03a9e9-aff6-4c95-ab84-882758a06b61" containerName="revoke-keys" Nov 02 21:50:32.580719 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.574058 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef03a9e9-aff6-4c95-ab84-882758a06b61" containerName="revoke-keys" Nov 02 21:50:32.580719 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.574523 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef03a9e9-aff6-4c95-ab84-882758a06b61" containerName="revoke-keys" Nov 02 21:50:32.582421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.582394 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.617538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.617510 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8flx\" (UniqueName: \"kubernetes.io/projected/ef03a9e9-aff6-4c95-ab84-882758a06b61-kube-api-access-w8flx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.617538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.617537 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef03a9e9-aff6-4c95-ab84-882758a06b61-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:32.633375 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.633347 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj"] Nov 02 21:50:32.715757 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.715728 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb"] Nov 02 21:50:32.718302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.718282 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.718367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.718319 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.718483 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.718461 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6r5\" (UniqueName: \"kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.719664 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.719651 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.776944 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.776869 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb"] Nov 02 21:50:32.819939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.819913 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.820090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.819949 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.820090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.819980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.820090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.820018 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6r5\" (UniqueName: \"kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.820198 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.820109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.820198 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.820141 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wztfc\" (UniqueName: \"kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.820752 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.820719 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.822224 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.822204 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.876493 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.876471 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6r5\" (UniqueName: \"kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5\") pod \"maas-api-revoke-sub-8684771a-kf5xj\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.898436 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.898402 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:32.921170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.921144 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.921268 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.921190 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wztfc\" (UniqueName: \"kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.921334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.921270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.921946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.921924 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.923461 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.923334 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:32.978322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:32.978279 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wztfc\" (UniqueName: \"kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc\") pod \"maas-api-revoke-sub-fcb1e7bf-kmnvb\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:33.032503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.032435 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:33.132064 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.132032 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5"] Nov 02 21:50:33.137273 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.137241 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.196705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.195271 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5"] Nov 02 21:50:33.225054 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.225025 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.225257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.225237 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.225302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.225290 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjd8w\" (UniqueName: \"kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.268880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.268847 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" event={"ID":"02c182a9-2d44-4204-a23e-064e717739bd","Type":"ContainerStarted","Data":"00cc9d8ef7eff175c895697026e7dd67f78a76de346da9905198c9d536583c58"} Nov 02 21:50:33.269021 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.268899 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" event={"ID":"02c182a9-2d44-4204-a23e-064e717739bd","Type":"ContainerStarted","Data":"83c7d3e1a24f14ad6fee1525356016159a2c3ac501363aca80791ba6fa477601"} Nov 02 21:50:33.270201 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.270182 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" Nov 02 21:50:33.270302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.270208 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s" event={"ID":"ef03a9e9-aff6-4c95-ab84-882758a06b61","Type":"ContainerDied","Data":"9469d550030978d65739fea8c63bac4598603d265dba88cc02f4332a93fdd4fe"} Nov 02 21:50:33.270302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.270245 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9469d550030978d65739fea8c63bac4598603d265dba88cc02f4332a93fdd4fe" Nov 02 21:50:33.274885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.274867 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj"] Nov 02 21:50:33.276374 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:33.276351 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba80a7e_6b85_4ce0_b6da_51faf3f51d1a.slice/crio-2d35daf64cf3109d5728286fa8c5f970c25a2debb1d15e0defbd065f7a90daeb WatchSource:0}: Error finding container 2d35daf64cf3109d5728286fa8c5f970c25a2debb1d15e0defbd065f7a90daeb: Status 404 returned error can't find the container with id 2d35daf64cf3109d5728286fa8c5f970c25a2debb1d15e0defbd065f7a90daeb Nov 02 21:50:33.326754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.326729 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.326861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.326781 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjd8w\" (UniqueName: \"kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.326861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.326822 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.327470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.327446 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.329109 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.329092 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.382097 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.382050 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" podStartSLOduration=2.382036671 podStartE2EDuration="2.382036671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:33.380165444 +0000 UTC m=+980.542260564" watchObservedRunningTime="2026-11-02 21:50:33.382036671 +0000 UTC m=+980.544131789" Nov 02 21:50:33.387695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.387653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjd8w\" (UniqueName: \"kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w\") pod \"maas-api-revoke-sub-48217aa8-thnn5\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.410929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.410908 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb"] Nov 02 21:50:33.411974 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:33.411952 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc12f73_e0fa_4455_abe4_a60413b4f161.slice/crio-b5c18c074092cb093938a31fcbf4f0af7c01c415ffb4cd9bf7426761956bf060 WatchSource:0}: Error finding container b5c18c074092cb093938a31fcbf4f0af7c01c415ffb4cd9bf7426761956bf060: Status 404 returned error can't find the container with id b5c18c074092cb093938a31fcbf4f0af7c01c415ffb4cd9bf7426761956bf060 Nov 02 21:50:33.452461 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.452440 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:33.826709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:33.826662 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5"] Nov 02 21:50:33.828150 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:33.828117 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856f63ae_c095_4def_ae81_961ba4f44bd7.slice/crio-3acb35951013fd3f1da63ae14e5d3886a8bc60c952bae4b4a7cb4b959a004430 WatchSource:0}: Error finding container 3acb35951013fd3f1da63ae14e5d3886a8bc60c952bae4b4a7cb4b959a004430: Status 404 returned error can't find the container with id 3acb35951013fd3f1da63ae14e5d3886a8bc60c952bae4b4a7cb4b959a004430 Nov 02 21:50:34.291499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.291345 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" event={"ID":"856f63ae-c095-4def-ae81-961ba4f44bd7","Type":"ContainerStarted","Data":"2affd7c3413314098602aca66a708bef1e2fb0356d8750a9822673d936d80cf3"} Nov 02 21:50:34.291499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.291399 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" event={"ID":"856f63ae-c095-4def-ae81-961ba4f44bd7","Type":"ContainerStarted","Data":"3acb35951013fd3f1da63ae14e5d3886a8bc60c952bae4b4a7cb4b959a004430"} Nov 02 21:50:34.293641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.293522 5091 generic.go:356] "Generic (PLEG): container finished" podID="02c182a9-2d44-4204-a23e-064e717739bd" containerID="00cc9d8ef7eff175c895697026e7dd67f78a76de346da9905198c9d536583c58" exitCode=0 Nov 02 21:50:34.293641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.293557 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" event={"ID":"02c182a9-2d44-4204-a23e-064e717739bd","Type":"ContainerDied","Data":"00cc9d8ef7eff175c895697026e7dd67f78a76de346da9905198c9d536583c58"} Nov 02 21:50:34.295439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.295412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" event={"ID":"8cc12f73-e0fa-4455-abe4-a60413b4f161","Type":"ContainerStarted","Data":"49645cc91813369acfeb3a0bb19880bbd5a5ca1d7c37c4ebf162e779ead4b8c9"} Nov 02 21:50:34.295550 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.295445 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" event={"ID":"8cc12f73-e0fa-4455-abe4-a60413b4f161","Type":"ContainerStarted","Data":"b5c18c074092cb093938a31fcbf4f0af7c01c415ffb4cd9bf7426761956bf060"} Nov 02 21:50:34.297113 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.297084 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" event={"ID":"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a","Type":"ContainerStarted","Data":"071a3563a912a55ddfe4c9ddc26f584032809b35f7f5278d392f6b1090eac2ff"} Nov 02 21:50:34.297197 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.297123 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" event={"ID":"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a","Type":"ContainerStarted","Data":"2d35daf64cf3109d5728286fa8c5f970c25a2debb1d15e0defbd065f7a90daeb"} Nov 02 21:50:34.405504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.405455 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" podStartSLOduration=1.405441843 podStartE2EDuration="1.405441843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:34.402928807 +0000 UTC m=+981.565023955" watchObservedRunningTime="2026-11-02 21:50:34.405441843 +0000 UTC m=+981.567536969" Nov 02 21:50:34.766708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:34.766636 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" podStartSLOduration=2.76662218 podStartE2EDuration="2.76662218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:34.763897782 +0000 UTC m=+981.925992906" watchObservedRunningTime="2026-11-02 21:50:34.76662218 +0000 UTC m=+981.928717297" Nov 02 21:50:35.306394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.306345 5091 generic.go:356] "Generic (PLEG): container finished" podID="856f63ae-c095-4def-ae81-961ba4f44bd7" containerID="2affd7c3413314098602aca66a708bef1e2fb0356d8750a9822673d936d80cf3" exitCode=0 Nov 02 21:50:35.306584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.306396 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" event={"ID":"856f63ae-c095-4def-ae81-961ba4f44bd7","Type":"ContainerDied","Data":"2affd7c3413314098602aca66a708bef1e2fb0356d8750a9822673d936d80cf3"} Nov 02 21:50:35.308419 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.308388 5091 generic.go:356] "Generic (PLEG): container finished" podID="8cc12f73-e0fa-4455-abe4-a60413b4f161" containerID="49645cc91813369acfeb3a0bb19880bbd5a5ca1d7c37c4ebf162e779ead4b8c9" exitCode=0 Nov 02 21:50:35.308543 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.308471 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" event={"ID":"8cc12f73-e0fa-4455-abe4-a60413b4f161","Type":"ContainerDied","Data":"49645cc91813369acfeb3a0bb19880bbd5a5ca1d7c37c4ebf162e779ead4b8c9"} Nov 02 21:50:35.310472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.310446 5091 generic.go:356] "Generic (PLEG): container finished" podID="dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" containerID="071a3563a912a55ddfe4c9ddc26f584032809b35f7f5278d392f6b1090eac2ff" exitCode=0 Nov 02 21:50:35.310576 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.310490 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" event={"ID":"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a","Type":"ContainerDied","Data":"071a3563a912a55ddfe4c9ddc26f584032809b35f7f5278d392f6b1090eac2ff"} Nov 02 21:50:35.511450 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.511422 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:35.544942 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.544901 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.651651 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca\") pod \"02c182a9-2d44-4204-a23e-064e717739bd\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.651719 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc\" (UniqueName: \"kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc\") pod \"8cc12f73-e0fa-4455-abe4-a60413b4f161\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.651774 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9\" (UniqueName: \"kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9\") pod \"02c182a9-2d44-4204-a23e-064e717739bd\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.651807 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca\") pod \"02c182a9-2d44-4204-a23e-064e717739bd\" (UID: \"02c182a9-2d44-4204-a23e-064e717739bd\") " Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.651893 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca\") pod \"8cc12f73-e0fa-4455-abe4-a60413b4f161\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.651950 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca\") pod \"8cc12f73-e0fa-4455-abe4-a60413b4f161\" (UID: \"8cc12f73-e0fa-4455-abe4-a60413b4f161\") " Nov 02 21:50:35.652971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.652932 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca" pod "8cc12f73-e0fa-4455-abe4-a60413b4f161" (UID: "8cc12f73-e0fa-4455-abe4-a60413b4f161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:35.660195 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.660158 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca" pod "02c182a9-2d44-4204-a23e-064e717739bd" (UID: "02c182a9-2d44-4204-a23e-064e717739bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:35.660556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.660525 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca" pod "02c182a9-2d44-4204-a23e-064e717739bd" (UID: "02c182a9-2d44-4204-a23e-064e717739bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:35.680934 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.679299 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9" pod "02c182a9-2d44-4204-a23e-064e717739bd" (UID: "02c182a9-2d44-4204-a23e-064e717739bd"). InnerVolumeSpecName "kube-api-access-xrrb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:35.681051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.680934 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca" pod "8cc12f73-e0fa-4455-abe4-a60413b4f161" (UID: "8cc12f73-e0fa-4455-abe4-a60413b4f161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:35.686822 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.686779 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc" pod "8cc12f73-e0fa-4455-abe4-a60413b4f161" (UID: "8cc12f73-e0fa-4455-abe4-a60413b4f161"). InnerVolumeSpecName "kube-api-access-wztfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:35.753834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.753794 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:35.753834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.753835 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:35.754037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.753851 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wztfc\" (UniqueName: \"kubernetes.io/projected/8cc12f73-e0fa-4455-abe4-a60413b4f161-kube-api-access-wztfc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:35.754037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.753866 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrb9\" (UniqueName: \"kubernetes.io/projected/02c182a9-2d44-4204-a23e-064e717739bd-kube-api-access-xrrb9\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:35.754037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.753881 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c182a9-2d44-4204-a23e-064e717739bd-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:35.754037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:35.753895 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc12f73-e0fa-4455-abe4-a60413b4f161-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:36.335699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.335246 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" event={"ID":"02c182a9-2d44-4204-a23e-064e717739bd","Type":"ContainerDied","Data":"83c7d3e1a24f14ad6fee1525356016159a2c3ac501363aca80791ba6fa477601"} Nov 02 21:50:36.335699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.335306 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c7d3e1a24f14ad6fee1525356016159a2c3ac501363aca80791ba6fa477601" Nov 02 21:50:36.335699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.335375 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5" Nov 02 21:50:36.338210 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.338185 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" Nov 02 21:50:36.339076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.338362 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb" event={"ID":"8cc12f73-e0fa-4455-abe4-a60413b4f161","Type":"ContainerDied","Data":"b5c18c074092cb093938a31fcbf4f0af7c01c415ffb4cd9bf7426761956bf060"} Nov 02 21:50:36.339076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.339052 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5c18c074092cb093938a31fcbf4f0af7c01c415ffb4cd9bf7426761956bf060" Nov 02 21:50:36.760205 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.760175 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:36.953295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.951140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5\" (UniqueName: \"kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5\") pod \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " Nov 02 21:50:36.953295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.951251 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca\") pod \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " Nov 02 21:50:36.953295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.951298 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca\") pod \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\" (UID: \"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a\") " Nov 02 21:50:36.959712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.956839 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca" pod "dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" (UID: "dba80a7e-6b85-4ce0-b6da-51faf3f51d1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:36.959712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.958936 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca" pod "dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" (UID: "dba80a7e-6b85-4ce0-b6da-51faf3f51d1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:36.962207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:36.962180 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5" pod "dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" (UID: "dba80a7e-6b85-4ce0-b6da-51faf3f51d1a"). InnerVolumeSpecName "kube-api-access-vj6r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:37.039484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.037107 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:37.052581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.052545 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj6r5\" (UniqueName: \"kubernetes.io/projected/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-kube-api-access-vj6r5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.052711 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.052583 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.052711 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.052599 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.153640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.153606 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w\" (UniqueName: \"kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w\") pod \"856f63ae-c095-4def-ae81-961ba4f44bd7\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " Nov 02 21:50:37.153838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.153660 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca\") pod \"856f63ae-c095-4def-ae81-961ba4f44bd7\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " Nov 02 21:50:37.153838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.153758 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca\") pod \"856f63ae-c095-4def-ae81-961ba4f44bd7\" (UID: \"856f63ae-c095-4def-ae81-961ba4f44bd7\") " Nov 02 21:50:37.154284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.154255 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca" pod "856f63ae-c095-4def-ae81-961ba4f44bd7" (UID: "856f63ae-c095-4def-ae81-961ba4f44bd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:37.156268 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.156243 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca" pod "856f63ae-c095-4def-ae81-961ba4f44bd7" (UID: "856f63ae-c095-4def-ae81-961ba4f44bd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:37.156486 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.156456 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w" pod "856f63ae-c095-4def-ae81-961ba4f44bd7" (UID: "856f63ae-c095-4def-ae81-961ba4f44bd7"). InnerVolumeSpecName "kube-api-access-bjd8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:37.194701 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:50:37.190027 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:50:37.255491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.255395 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjd8w\" (UniqueName: \"kubernetes.io/projected/856f63ae-c095-4def-ae81-961ba4f44bd7-kube-api-access-bjd8w\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.255491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.255440 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.255491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.255457 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856f63ae-c095-4def-ae81-961ba4f44bd7-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.349494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.349469 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" Nov 02 21:50:37.349682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.349516 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj" event={"ID":"dba80a7e-6b85-4ce0-b6da-51faf3f51d1a","Type":"ContainerDied","Data":"2d35daf64cf3109d5728286fa8c5f970c25a2debb1d15e0defbd065f7a90daeb"} Nov 02 21:50:37.349682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.349570 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d35daf64cf3109d5728286fa8c5f970c25a2debb1d15e0defbd065f7a90daeb" Nov 02 21:50:37.351163 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.351145 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" Nov 02 21:50:37.351302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.351186 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5" event={"ID":"856f63ae-c095-4def-ae81-961ba4f44bd7","Type":"ContainerDied","Data":"3acb35951013fd3f1da63ae14e5d3886a8bc60c952bae4b4a7cb4b959a004430"} Nov 02 21:50:37.351302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:37.351226 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3acb35951013fd3f1da63ae14e5d3886a8bc60c952bae4b4a7cb4b959a004430" Nov 02 21:50:42.524428 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524394 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl"] Nov 02 21:50:42.524887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524873 5091 cpu_manager.go:414] "containerMap: removing container" podUID="02c182a9-2d44-4204-a23e-064e717739bd" containerName="revoke-keys" Nov 02 21:50:42.524928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524888 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c182a9-2d44-4204-a23e-064e717739bd" containerName="revoke-keys" Nov 02 21:50:42.524928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524898 5091 cpu_manager.go:414] "containerMap: removing container" podUID="dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" containerName="revoke-keys" Nov 02 21:50:42.524928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524903 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" containerName="revoke-keys" Nov 02 21:50:42.524928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524918 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8cc12f73-e0fa-4455-abe4-a60413b4f161" containerName="revoke-keys" Nov 02 21:50:42.524928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524924 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc12f73-e0fa-4455-abe4-a60413b4f161" containerName="revoke-keys" Nov 02 21:50:42.525072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524937 5091 cpu_manager.go:414] "containerMap: removing container" podUID="856f63ae-c095-4def-ae81-961ba4f44bd7" containerName="revoke-keys" Nov 02 21:50:42.525072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.524942 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856f63ae-c095-4def-ae81-961ba4f44bd7" containerName="revoke-keys" Nov 02 21:50:42.525072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.525025 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc12f73-e0fa-4455-abe4-a60413b4f161" containerName="revoke-keys" Nov 02 21:50:42.525072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.525033 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="856f63ae-c095-4def-ae81-961ba4f44bd7" containerName="revoke-keys" Nov 02 21:50:42.525072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.525043 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c182a9-2d44-4204-a23e-064e717739bd" containerName="revoke-keys" Nov 02 21:50:42.525072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.525049 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" containerName="revoke-keys" Nov 02 21:50:42.528526 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.528509 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.582115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.582090 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:42.605765 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.605738 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.605891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.605773 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqr89\" (UniqueName: \"kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.605891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.605794 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.631585 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.631559 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:42.631713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.631608 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:50:42.631713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.631703 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:42.634710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.634687 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl"] Nov 02 21:50:42.706963 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.706940 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.707090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.706972 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqr89\" (UniqueName: \"kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.707090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.706991 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.707782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.707762 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.709542 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.709518 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.762900 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.762872 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqr89\" (UniqueName: \"kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89\") pod \"maas-api-revoke-sub-603095a0-cc4hl\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:42.842104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:42.842029 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:43.213449 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:43.213409 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl"] Nov 02 21:50:43.214180 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:43.214151 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad54f272_861e_45fe_ab4b_525ce8a61713.slice/crio-b3a0b6672dcfef5b4c690cc7d12cb3e22477d0849638bfddcf6ce2470ed887ca WatchSource:0}: Error finding container b3a0b6672dcfef5b4c690cc7d12cb3e22477d0849638bfddcf6ce2470ed887ca: Status 404 returned error can't find the container with id b3a0b6672dcfef5b4c690cc7d12cb3e22477d0849638bfddcf6ce2470ed887ca Nov 02 21:50:43.388289 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:43.388253 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" event={"ID":"ad54f272-861e-45fe-ab4b-525ce8a61713","Type":"ContainerStarted","Data":"b3a0b6672dcfef5b4c690cc7d12cb3e22477d0849638bfddcf6ce2470ed887ca"} Nov 02 21:50:44.395849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:44.395771 5091 generic.go:356] "Generic (PLEG): container finished" podID="ad54f272-861e-45fe-ab4b-525ce8a61713" containerID="05d5eb7696bd982c58b531da0739b2f73ebb1f8a53b678a7c148ee36261170a1" exitCode=0 Nov 02 21:50:44.396207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:44.395853 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" event={"ID":"ad54f272-861e-45fe-ab4b-525ce8a61713","Type":"ContainerDied","Data":"05d5eb7696bd982c58b531da0739b2f73ebb1f8a53b678a7c148ee36261170a1"} Nov 02 21:50:45.548193 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.548170 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:45.637022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.636992 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca\") pod \"ad54f272-861e-45fe-ab4b-525ce8a61713\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " Nov 02 21:50:45.637210 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.637042 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca\") pod \"ad54f272-861e-45fe-ab4b-525ce8a61713\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " Nov 02 21:50:45.637210 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.637153 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89\" (UniqueName: \"kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89\") pod \"ad54f272-861e-45fe-ab4b-525ce8a61713\" (UID: \"ad54f272-861e-45fe-ab4b-525ce8a61713\") " Nov 02 21:50:45.637441 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.637405 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca" pod "ad54f272-861e-45fe-ab4b-525ce8a61713" (UID: "ad54f272-861e-45fe-ab4b-525ce8a61713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:45.639213 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.639185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca" pod "ad54f272-861e-45fe-ab4b-525ce8a61713" (UID: "ad54f272-861e-45fe-ab4b-525ce8a61713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:45.639491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.639474 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89" pod "ad54f272-861e-45fe-ab4b-525ce8a61713" (UID: "ad54f272-861e-45fe-ab4b-525ce8a61713"). InnerVolumeSpecName "kube-api-access-tqr89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:45.738257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.738189 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:45.738257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.738216 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad54f272-861e-45fe-ab4b-525ce8a61713-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:45.738257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:45.738225 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqr89\" (UniqueName: \"kubernetes.io/projected/ad54f272-861e-45fe-ab4b-525ce8a61713-kube-api-access-tqr89\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:46.224453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.224421 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4"] Nov 02 21:50:46.224979 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.224964 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ad54f272-861e-45fe-ab4b-525ce8a61713" containerName="revoke-keys" Nov 02 21:50:46.225034 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.224980 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad54f272-861e-45fe-ab4b-525ce8a61713" containerName="revoke-keys" Nov 02 21:50:46.225102 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.225092 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad54f272-861e-45fe-ab4b-525ce8a61713" containerName="revoke-keys" Nov 02 21:50:46.228395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.228375 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.242287 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.242260 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.242421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.242309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf62v\" (UniqueName: \"kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.242583 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.242488 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.285010 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.284978 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4"] Nov 02 21:50:46.343285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.343253 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf62v\" (UniqueName: \"kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.343446 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.343371 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.343446 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.343418 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.344154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.344131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.345764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.345745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.399062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.399032 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf62v\" (UniqueName: \"kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v\") pod \"maas-api-revoke-sub-dfcd6548-jpfd4\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.409730 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.409704 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" event={"ID":"ad54f272-861e-45fe-ab4b-525ce8a61713","Type":"ContainerDied","Data":"b3a0b6672dcfef5b4c690cc7d12cb3e22477d0849638bfddcf6ce2470ed887ca"} Nov 02 21:50:46.409855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.409760 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3a0b6672dcfef5b4c690cc7d12cb3e22477d0849638bfddcf6ce2470ed887ca" Nov 02 21:50:46.409855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.409718 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl" Nov 02 21:50:46.543203 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.543128 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:46.915421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:46.915397 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4"] Nov 02 21:50:46.917528 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:46.917493 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466a7651_8cbf_46a0_a236_8a6e25c09522.slice/crio-03926e4dabdc82523bf3313dd340c36f8a1d4bc6becf9803363a4162d1a0cd24 WatchSource:0}: Error finding container 03926e4dabdc82523bf3313dd340c36f8a1d4bc6becf9803363a4162d1a0cd24: Status 404 returned error can't find the container with id 03926e4dabdc82523bf3313dd340c36f8a1d4bc6becf9803363a4162d1a0cd24 Nov 02 21:50:47.417479 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:47.417435 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" event={"ID":"466a7651-8cbf-46a0-a236-8a6e25c09522","Type":"ContainerStarted","Data":"ee5bee8ba6f4333de954033f6863ace3f9e4c25d94a3884d9e7300abbc2ee1d7"} Nov 02 21:50:47.417662 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:47.417538 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" event={"ID":"466a7651-8cbf-46a0-a236-8a6e25c09522","Type":"ContainerStarted","Data":"03926e4dabdc82523bf3313dd340c36f8a1d4bc6becf9803363a4162d1a0cd24"} Nov 02 21:50:47.532027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:47.531934 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" podStartSLOduration=1.531921589 podStartE2EDuration="1.531921589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:47.528453404 +0000 UTC m=+994.690548525" watchObservedRunningTime="2026-11-02 21:50:47.531921589 +0000 UTC m=+994.694016706" Nov 02 21:50:48.297831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.297797 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9"] Nov 02 21:50:48.301659 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.301643 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.357808 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.357778 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9"] Nov 02 21:50:48.361965 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.361941 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.362106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.361987 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.362170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.362147 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2d9x\" (UniqueName: \"kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.425331 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.425299 5091 generic.go:356] "Generic (PLEG): container finished" podID="466a7651-8cbf-46a0-a236-8a6e25c09522" containerID="ee5bee8ba6f4333de954033f6863ace3f9e4c25d94a3884d9e7300abbc2ee1d7" exitCode=0 Nov 02 21:50:48.425473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.425345 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" event={"ID":"466a7651-8cbf-46a0-a236-8a6e25c09522","Type":"ContainerDied","Data":"ee5bee8ba6f4333de954033f6863ace3f9e4c25d94a3884d9e7300abbc2ee1d7"} Nov 02 21:50:48.463540 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.463514 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2d9x\" (UniqueName: \"kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.463699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.463565 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.463699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.463594 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.464211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.464186 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.465890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.465866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.519504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.519474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2d9x\" (UniqueName: \"kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x\") pod \"maas-api-revoke-sub-2c2249c8-6fxt9\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.614766 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.614664 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:48.984379 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:48.984347 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9"] Nov 02 21:50:48.984932 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:48.984905 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6476bc3f_9806_4f7e_8a00_5f4e53f2562e.slice/crio-a49f1f5eb3be4d926d099623799b6f394b3966bc45bb024553d5f2dd45d30149 WatchSource:0}: Error finding container a49f1f5eb3be4d926d099623799b6f394b3966bc45bb024553d5f2dd45d30149: Status 404 returned error can't find the container with id a49f1f5eb3be4d926d099623799b6f394b3966bc45bb024553d5f2dd45d30149 Nov 02 21:50:49.433096 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.433056 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" event={"ID":"6476bc3f-9806-4f7e-8a00-5f4e53f2562e","Type":"ContainerStarted","Data":"7ccda80562440704fc6bae441abed8c84ebc3a790a109e3ff17787d80e807e63"} Nov 02 21:50:49.433486 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.433122 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" event={"ID":"6476bc3f-9806-4f7e-8a00-5f4e53f2562e","Type":"ContainerStarted","Data":"a49f1f5eb3be4d926d099623799b6f394b3966bc45bb024553d5f2dd45d30149"} Nov 02 21:50:49.549159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.549073 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" podStartSLOduration=1.549056972 podStartE2EDuration="1.549056972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:49.545935681 +0000 UTC m=+996.708030802" watchObservedRunningTime="2026-11-02 21:50:49.549056972 +0000 UTC m=+996.711152135" Nov 02 21:50:49.577309 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.577285 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:49.675285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.675253 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v\" (UniqueName: \"kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v\") pod \"466a7651-8cbf-46a0-a236-8a6e25c09522\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " Nov 02 21:50:49.675473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.675390 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca\") pod \"466a7651-8cbf-46a0-a236-8a6e25c09522\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " Nov 02 21:50:49.675473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.675453 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca\") pod \"466a7651-8cbf-46a0-a236-8a6e25c09522\" (UID: \"466a7651-8cbf-46a0-a236-8a6e25c09522\") " Nov 02 21:50:49.675852 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.675823 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca" pod "466a7651-8cbf-46a0-a236-8a6e25c09522" (UID: "466a7651-8cbf-46a0-a236-8a6e25c09522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:49.677305 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.677280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v" pod "466a7651-8cbf-46a0-a236-8a6e25c09522" (UID: "466a7651-8cbf-46a0-a236-8a6e25c09522"). InnerVolumeSpecName "kube-api-access-jf62v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:49.677544 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.677527 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca" pod "466a7651-8cbf-46a0-a236-8a6e25c09522" (UID: "466a7651-8cbf-46a0-a236-8a6e25c09522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:49.776456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.776420 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf62v\" (UniqueName: \"kubernetes.io/projected/466a7651-8cbf-46a0-a236-8a6e25c09522-kube-api-access-jf62v\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:49.776456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.776456 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:49.776659 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:49.776472 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466a7651-8cbf-46a0-a236-8a6e25c09522-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:50.202277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.202244 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q"] Nov 02 21:50:50.202806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.202792 5091 cpu_manager.go:414] "containerMap: removing container" podUID="466a7651-8cbf-46a0-a236-8a6e25c09522" containerName="revoke-keys" Nov 02 21:50:50.202855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.202808 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466a7651-8cbf-46a0-a236-8a6e25c09522" containerName="revoke-keys" Nov 02 21:50:50.202893 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.202887 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="466a7651-8cbf-46a0-a236-8a6e25c09522" containerName="revoke-keys" Nov 02 21:50:50.206257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.206236 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.262855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.262825 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q"] Nov 02 21:50:50.281023 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.280998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkh7k\" (UniqueName: \"kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.281151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.281129 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.281192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.281185 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.381821 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.381791 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.382037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.381854 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.382037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.381881 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkh7k\" (UniqueName: \"kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.382422 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.382400 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.384132 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.384110 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.438018 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.437987 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkh7k\" (UniqueName: \"kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k\") pod \"maas-api-revoke-sub-6b67a2a3-kr95q\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.441476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.441446 5091 generic.go:356] "Generic (PLEG): container finished" podID="6476bc3f-9806-4f7e-8a00-5f4e53f2562e" containerID="7ccda80562440704fc6bae441abed8c84ebc3a790a109e3ff17787d80e807e63" exitCode=0 Nov 02 21:50:50.441570 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.441531 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" event={"ID":"6476bc3f-9806-4f7e-8a00-5f4e53f2562e","Type":"ContainerDied","Data":"7ccda80562440704fc6bae441abed8c84ebc3a790a109e3ff17787d80e807e63"} Nov 02 21:50:50.442874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.442850 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" event={"ID":"466a7651-8cbf-46a0-a236-8a6e25c09522","Type":"ContainerDied","Data":"03926e4dabdc82523bf3313dd340c36f8a1d4bc6becf9803363a4162d1a0cd24"} Nov 02 21:50:50.442874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.442865 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4" Nov 02 21:50:50.442992 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.442894 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03926e4dabdc82523bf3313dd340c36f8a1d4bc6becf9803363a4162d1a0cd24" Nov 02 21:50:50.520315 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.520243 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:50.894465 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:50.894433 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q"] Nov 02 21:50:50.896084 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:50.896049 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ba28f6_cd88_4006_9f20_21207be10eab.slice/crio-cb4ea71fd1a1a1c17f31639ea581b1d3515309628159d8331fd467f1eca353b5 WatchSource:0}: Error finding container cb4ea71fd1a1a1c17f31639ea581b1d3515309628159d8331fd467f1eca353b5: Status 404 returned error can't find the container with id cb4ea71fd1a1a1c17f31639ea581b1d3515309628159d8331fd467f1eca353b5 Nov 02 21:50:51.452695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.452584 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" event={"ID":"24ba28f6-cd88-4006-9f20-21207be10eab","Type":"ContainerStarted","Data":"b5c263486880341112ed7c1de0d6eceb88ef3a6a2659a8d5ad4d7dd844092fad"} Nov 02 21:50:51.452695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.452643 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" event={"ID":"24ba28f6-cd88-4006-9f20-21207be10eab","Type":"ContainerStarted","Data":"cb4ea71fd1a1a1c17f31639ea581b1d3515309628159d8331fd467f1eca353b5"} Nov 02 21:50:51.568516 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.568467 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" podStartSLOduration=1.568447726 podStartE2EDuration="1.568447726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:51.565864727 +0000 UTC m=+998.727959847" watchObservedRunningTime="2026-11-02 21:50:51.568447726 +0000 UTC m=+998.730542848" Nov 02 21:50:51.613294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.613270 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:51.698509 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.698476 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca\") pod \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " Nov 02 21:50:51.698703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.698657 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca\") pod \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " Nov 02 21:50:51.698801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.698783 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x\" (UniqueName: \"kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x\") pod \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\" (UID: \"6476bc3f-9806-4f7e-8a00-5f4e53f2562e\") " Nov 02 21:50:51.698912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.698893 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca" pod "6476bc3f-9806-4f7e-8a00-5f4e53f2562e" (UID: "6476bc3f-9806-4f7e-8a00-5f4e53f2562e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:51.699111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.699095 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:51.700758 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.700731 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x" pod "6476bc3f-9806-4f7e-8a00-5f4e53f2562e" (UID: "6476bc3f-9806-4f7e-8a00-5f4e53f2562e"). InnerVolumeSpecName "kube-api-access-b2d9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:51.700888 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.700865 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca" pod "6476bc3f-9806-4f7e-8a00-5f4e53f2562e" (UID: "6476bc3f-9806-4f7e-8a00-5f4e53f2562e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:51.800402 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.800376 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:51.800402 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:51.800401 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2d9x\" (UniqueName: \"kubernetes.io/projected/6476bc3f-9806-4f7e-8a00-5f4e53f2562e-kube-api-access-b2d9x\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:52.466692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:52.463526 5091 generic.go:356] "Generic (PLEG): container finished" podID="24ba28f6-cd88-4006-9f20-21207be10eab" containerID="b5c263486880341112ed7c1de0d6eceb88ef3a6a2659a8d5ad4d7dd844092fad" exitCode=0 Nov 02 21:50:52.466692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:52.463647 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" event={"ID":"24ba28f6-cd88-4006-9f20-21207be10eab","Type":"ContainerDied","Data":"b5c263486880341112ed7c1de0d6eceb88ef3a6a2659a8d5ad4d7dd844092fad"} Nov 02 21:50:52.466692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:52.466015 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" event={"ID":"6476bc3f-9806-4f7e-8a00-5f4e53f2562e","Type":"ContainerDied","Data":"a49f1f5eb3be4d926d099623799b6f394b3966bc45bb024553d5f2dd45d30149"} Nov 02 21:50:52.466692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:52.466069 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49f1f5eb3be4d926d099623799b6f394b3966bc45bb024553d5f2dd45d30149" Nov 02 21:50:52.466692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:52.466145 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9" Nov 02 21:50:53.642811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.642788 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:53.720735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.720702 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca\") pod \"24ba28f6-cd88-4006-9f20-21207be10eab\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " Nov 02 21:50:53.720939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.720749 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca\") pod \"24ba28f6-cd88-4006-9f20-21207be10eab\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " Nov 02 21:50:53.720939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.720916 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k\" (UniqueName: \"kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k\") pod \"24ba28f6-cd88-4006-9f20-21207be10eab\" (UID: \"24ba28f6-cd88-4006-9f20-21207be10eab\") " Nov 02 21:50:53.721758 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.721727 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca" pod "24ba28f6-cd88-4006-9f20-21207be10eab" (UID: "24ba28f6-cd88-4006-9f20-21207be10eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:53.723750 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.723718 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k" pod "24ba28f6-cd88-4006-9f20-21207be10eab" (UID: "24ba28f6-cd88-4006-9f20-21207be10eab"). InnerVolumeSpecName "kube-api-access-zkh7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:53.723985 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.723964 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca" pod "24ba28f6-cd88-4006-9f20-21207be10eab" (UID: "24ba28f6-cd88-4006-9f20-21207be10eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:53.823135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.823097 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:53.823135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.823134 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ba28f6-cd88-4006-9f20-21207be10eab-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:53.823388 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:53.823151 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkh7k\" (UniqueName: \"kubernetes.io/projected/24ba28f6-cd88-4006-9f20-21207be10eab-kube-api-access-zkh7k\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:54.482788 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:54.482754 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" event={"ID":"24ba28f6-cd88-4006-9f20-21207be10eab","Type":"ContainerDied","Data":"cb4ea71fd1a1a1c17f31639ea581b1d3515309628159d8331fd467f1eca353b5"} Nov 02 21:50:54.482974 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:54.482825 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4ea71fd1a1a1c17f31639ea581b1d3515309628159d8331fd467f1eca353b5" Nov 02 21:50:54.482974 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:54.482895 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q" Nov 02 21:50:56.930752 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.930719 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx"] Nov 02 21:50:56.931284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.931270 5091 cpu_manager.go:414] "containerMap: removing container" podUID="24ba28f6-cd88-4006-9f20-21207be10eab" containerName="revoke-keys" Nov 02 21:50:56.931340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.931284 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ba28f6-cd88-4006-9f20-21207be10eab" containerName="revoke-keys" Nov 02 21:50:56.931340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.931328 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6476bc3f-9806-4f7e-8a00-5f4e53f2562e" containerName="revoke-keys" Nov 02 21:50:56.931340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.931334 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6476bc3f-9806-4f7e-8a00-5f4e53f2562e" containerName="revoke-keys" Nov 02 21:50:56.931438 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.931411 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6476bc3f-9806-4f7e-8a00-5f4e53f2562e" containerName="revoke-keys" Nov 02 21:50:56.931438 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.931423 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ba28f6-cd88-4006-9f20-21207be10eab" containerName="revoke-keys" Nov 02 21:50:56.934905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.934886 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:56.991720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:56.991658 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx"] Nov 02 21:50:57.037434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.037400 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:57.037434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.037402 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:50:57.037619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.037402 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:50:57.037619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.037400 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:50:57.056252 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.056226 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.056341 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.056304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsp2\" (UniqueName: \"kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.056431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.056415 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.157575 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.157543 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsp2\" (UniqueName: \"kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.157745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.157621 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.157745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.157731 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.158360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.158336 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.159936 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.159913 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.225488 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.225411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsp2\" (UniqueName: \"kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2\") pod \"maas-api-revoke-sub-9adbb28f-sp5vx\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.249335 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.249313 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:50:57.621895 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:57.621869 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx"] Nov 02 21:50:57.622362 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:57.622333 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3a4ffc_6a33_4bf4_aaca_0944458a9084.slice/crio-11c8aa458484c7afe97a1d893ab3eba613b0a9a70c859e891ccab3ee08abe377 WatchSource:0}: Error finding container 11c8aa458484c7afe97a1d893ab3eba613b0a9a70c859e891ccab3ee08abe377: Status 404 returned error can't find the container with id 11c8aa458484c7afe97a1d893ab3eba613b0a9a70c859e891ccab3ee08abe377 Nov 02 21:50:58.512414 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.512376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" event={"ID":"bb3a4ffc-6a33-4bf4-aaca-0944458a9084","Type":"ContainerStarted","Data":"e6f79d82bbe0126795b19e80d1d637816bbdbf4e56eed271a20eaf41e9b4fb2b"} Nov 02 21:50:58.512871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.512433 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" event={"ID":"bb3a4ffc-6a33-4bf4-aaca-0944458a9084","Type":"ContainerStarted","Data":"11c8aa458484c7afe97a1d893ab3eba613b0a9a70c859e891ccab3ee08abe377"} Nov 02 21:50:58.630735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.630660 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" podStartSLOduration=2.630642193 podStartE2EDuration="2.630642193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:58.629541387 +0000 UTC m=+1005.791636506" watchObservedRunningTime="2026-11-02 21:50:58.630642193 +0000 UTC m=+1005.792737313" Nov 02 21:50:58.698321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.698291 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt"] Nov 02 21:50:58.702270 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.702253 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.749573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.749548 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6pbm\" (UniqueName: \"kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.749698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.749595 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.749782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.749756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.759248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.759222 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt"] Nov 02 21:50:58.851088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.851008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.851088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.851069 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6pbm\" (UniqueName: \"kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.851304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.851119 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.851765 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.851741 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.853402 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.853382 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:58.909073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:58.909046 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6pbm\" (UniqueName: \"kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm\") pod \"maas-api-revoke-sub-a97389dd-d2fnt\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:59.015087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:59.015057 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:50:59.388389 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:59.388359 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt"] Nov 02 21:50:59.388729 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:50:59.388702 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcbdbb3_4a8d_45bc_b0a1_f3ff5ae536e2.slice/crio-649292b4639d757824bbe0114b19b36f96b31d58a50517bd71ca63cdbded3573 WatchSource:0}: Error finding container 649292b4639d757824bbe0114b19b36f96b31d58a50517bd71ca63cdbded3573: Status 404 returned error can't find the container with id 649292b4639d757824bbe0114b19b36f96b31d58a50517bd71ca63cdbded3573 Nov 02 21:50:59.520354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:59.520323 5091 generic.go:356] "Generic (PLEG): container finished" podID="bb3a4ffc-6a33-4bf4-aaca-0944458a9084" containerID="e6f79d82bbe0126795b19e80d1d637816bbdbf4e56eed271a20eaf41e9b4fb2b" exitCode=0 Nov 02 21:50:59.520776 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:59.520364 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" event={"ID":"bb3a4ffc-6a33-4bf4-aaca-0944458a9084","Type":"ContainerDied","Data":"e6f79d82bbe0126795b19e80d1d637816bbdbf4e56eed271a20eaf41e9b4fb2b"} Nov 02 21:50:59.521780 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:50:59.521754 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" event={"ID":"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2","Type":"ContainerStarted","Data":"649292b4639d757824bbe0114b19b36f96b31d58a50517bd71ca63cdbded3573"} Nov 02 21:51:00.529564 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.529487 5091 generic.go:356] "Generic (PLEG): container finished" podID="0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" containerID="813767329ec08aa9ad332676c3bcb44fafcb6b422a713a061555387e46c98ec1" exitCode=0 Nov 02 21:51:00.530038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.529579 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" event={"ID":"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2","Type":"ContainerDied","Data":"813767329ec08aa9ad332676c3bcb44fafcb6b422a713a061555387e46c98ec1"} Nov 02 21:51:00.684365 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.684337 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:51:00.771734 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.771692 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2\" (UniqueName: \"kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2\") pod \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " Nov 02 21:51:00.771878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.771750 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca\") pod \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " Nov 02 21:51:00.771878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.771830 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca\") pod \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\" (UID: \"bb3a4ffc-6a33-4bf4-aaca-0944458a9084\") " Nov 02 21:51:00.772150 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.772121 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca" pod "bb3a4ffc-6a33-4bf4-aaca-0944458a9084" (UID: "bb3a4ffc-6a33-4bf4-aaca-0944458a9084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:00.773842 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.773821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca" pod "bb3a4ffc-6a33-4bf4-aaca-0944458a9084" (UID: "bb3a4ffc-6a33-4bf4-aaca-0944458a9084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:00.773923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.773821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2" pod "bb3a4ffc-6a33-4bf4-aaca-0944458a9084" (UID: "bb3a4ffc-6a33-4bf4-aaca-0944458a9084"). InnerVolumeSpecName "kube-api-access-qdsp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:00.873308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.873269 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:00.873308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.873307 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdsp2\" (UniqueName: \"kubernetes.io/projected/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-kube-api-access-qdsp2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:00.873308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:00.873318 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a4ffc-6a33-4bf4-aaca-0944458a9084-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:01.537523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.537488 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" Nov 02 21:51:01.537929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.537519 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx" event={"ID":"bb3a4ffc-6a33-4bf4-aaca-0944458a9084","Type":"ContainerDied","Data":"11c8aa458484c7afe97a1d893ab3eba613b0a9a70c859e891ccab3ee08abe377"} Nov 02 21:51:01.537929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.537573 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c8aa458484c7afe97a1d893ab3eba613b0a9a70c859e891ccab3ee08abe377" Nov 02 21:51:01.681224 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.681200 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:51:01.781444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.781404 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca\") pod \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " Nov 02 21:51:01.781627 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.781516 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm\" (UniqueName: \"kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm\") pod \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " Nov 02 21:51:01.781627 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.781546 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca\") pod \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\" (UID: \"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2\") " Nov 02 21:51:01.781836 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.781806 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca" pod "0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" (UID: "0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:01.781986 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.781971 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:01.783651 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.783626 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm" pod "0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" (UID: "0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2"). InnerVolumeSpecName "kube-api-access-f6pbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:01.783791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.783631 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca" pod "0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" (UID: "0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:01.883448 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.883416 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6pbm\" (UniqueName: \"kubernetes.io/projected/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-kube-api-access-f6pbm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:01.883448 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:01.883442 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:02.545009 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:02.544971 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" Nov 02 21:51:02.545401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:02.545008 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt" event={"ID":"0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2","Type":"ContainerDied","Data":"649292b4639d757824bbe0114b19b36f96b31d58a50517bd71ca63cdbded3573"} Nov 02 21:51:02.545401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:02.545064 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="649292b4639d757824bbe0114b19b36f96b31d58a50517bd71ca63cdbded3573" Nov 02 21:51:06.090433 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.090393 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2"] Nov 02 21:51:06.090934 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.090919 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bb3a4ffc-6a33-4bf4-aaca-0944458a9084" containerName="revoke-keys" Nov 02 21:51:06.090977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.090938 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3a4ffc-6a33-4bf4-aaca-0944458a9084" containerName="revoke-keys" Nov 02 21:51:06.090977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.090951 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" containerName="revoke-keys" Nov 02 21:51:06.090977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.090956 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" containerName="revoke-keys" Nov 02 21:51:06.091068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.091043 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3a4ffc-6a33-4bf4-aaca-0944458a9084" containerName="revoke-keys" Nov 02 21:51:06.091068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.091053 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" containerName="revoke-keys" Nov 02 21:51:06.095821 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.095801 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.150732 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.150701 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2"] Nov 02 21:51:06.196558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.196529 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:51:06.196558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.196541 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:51:06.197618 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.197602 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:51:06.197735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.197719 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:51:06.237883 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.237852 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9pl\" (UniqueName: \"kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.237883 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.237884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.238055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.237914 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.338566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.338536 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9pl\" (UniqueName: \"kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.338566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.338569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.338770 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.338600 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.339256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.339239 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.340935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.340891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.395578 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.395551 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9pl\" (UniqueName: \"kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl\") pod \"maas-api-revoke-sub-4e799f06-m7mn2\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.409106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.409087 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:06.780316 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:06.780282 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07e55ce_ad20_4045_bbc6_ebe66bb80ec3.slice/crio-e25df62fd1e378ccc7703c1e2bfc30b527eeed8553086d786677bbe80af82d3d WatchSource:0}: Error finding container e25df62fd1e378ccc7703c1e2bfc30b527eeed8553086d786677bbe80af82d3d: Status 404 returned error can't find the container with id e25df62fd1e378ccc7703c1e2bfc30b527eeed8553086d786677bbe80af82d3d Nov 02 21:51:06.781311 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:06.781283 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2"] Nov 02 21:51:07.110833 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:51:07.110798 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:51:07.580817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:07.580791 5091 generic.go:356] "Generic (PLEG): container finished" podID="f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" containerID="d943e191a67ac044abece2437503a90501d68216f90e9fb71cc53429a8bbb930" exitCode=0 Nov 02 21:51:07.580933 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:07.580875 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" event={"ID":"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3","Type":"ContainerDied","Data":"d943e191a67ac044abece2437503a90501d68216f90e9fb71cc53429a8bbb930"} Nov 02 21:51:07.580986 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:07.580940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" event={"ID":"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3","Type":"ContainerStarted","Data":"e25df62fd1e378ccc7703c1e2bfc30b527eeed8553086d786677bbe80af82d3d"} Nov 02 21:51:08.728975 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.728948 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:08.864354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.864271 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca\") pod \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " Nov 02 21:51:08.864354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.864330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca\") pod \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " Nov 02 21:51:08.864579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.864397 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl\" (UniqueName: \"kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl\") pod \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\" (UID: \"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3\") " Nov 02 21:51:08.864826 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.864791 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca" pod "f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" (UID: "f07e55ce-ad20-4045-bbc6-ebe66bb80ec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:08.866496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.866465 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca" pod "f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" (UID: "f07e55ce-ad20-4045-bbc6-ebe66bb80ec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:08.866590 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.866500 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl" pod "f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" (UID: "f07e55ce-ad20-4045-bbc6-ebe66bb80ec3"). InnerVolumeSpecName "kube-api-access-8c9pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:08.966233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.966205 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:08.966356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.966236 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c9pl\" (UniqueName: \"kubernetes.io/projected/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-kube-api-access-8c9pl\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:08.966356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:08.966249 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:09.593811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:09.593778 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" Nov 02 21:51:09.593977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:09.593810 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2" event={"ID":"f07e55ce-ad20-4045-bbc6-ebe66bb80ec3","Type":"ContainerDied","Data":"e25df62fd1e378ccc7703c1e2bfc30b527eeed8553086d786677bbe80af82d3d"} Nov 02 21:51:09.593977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:09.593880 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25df62fd1e378ccc7703c1e2bfc30b527eeed8553086d786677bbe80af82d3d" Nov 02 21:51:10.525481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.525442 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8"] Nov 02 21:51:10.526284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.526257 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" containerName="revoke-keys" Nov 02 21:51:10.526344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.526285 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" containerName="revoke-keys" Nov 02 21:51:10.526429 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.526415 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" containerName="revoke-keys" Nov 02 21:51:10.530577 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.530551 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.639222 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.639189 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8"] Nov 02 21:51:10.672913 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.672884 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9cc56b-openshift-default-dockercfg-hl779\"" Nov 02 21:51:10.689347 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689313 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmgx\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689356 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689444 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689503 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689573 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689596 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689626 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.689796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.689777 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790379 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmgx\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790379 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790335 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790599 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790382 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790599 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790414 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790599 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790448 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790599 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790647 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790770 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790803 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790852 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.790962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790897 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.791053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.790976 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.791053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.791008 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.791255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.791220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.793148 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.793125 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.793391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.793370 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.849162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.849129 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmgx\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.849524 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.849503 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:10.864198 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.864153 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h"] Nov 02 21:51:10.869549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.869524 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.923468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.923435 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f48c4d-openshift-default-dockercfg-xpv97\"" Nov 02 21:51:10.928761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.928727 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h"] Nov 02 21:51:10.993143 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993105 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993160 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993244 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993280 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993313 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993391 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993514 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:10.993582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:10.993570 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj92b\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.094798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.094650 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj92b\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.094978 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.094830 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.094978 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.094875 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.094978 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.094915 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.094978 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.094955 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.094996 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095025 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095176 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095469 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095377 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095870 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095990 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095947 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.095990 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095967 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.096118 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.095996 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.098039 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.098016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.098329 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.098308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.144882 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.144851 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:11.151340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.151309 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.151496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.151474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj92b\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.184755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.184719 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:11.641524 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.641031 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8"] Nov 02 21:51:11.643796 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:11.643754 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28033c5c_b32b_4d67_8326_bf25ca1afee4.slice/crio-cef51de9f06232b2e7f69e719dc383346173be45ce435254c2870199be93c305 WatchSource:0}: Error finding container cef51de9f06232b2e7f69e719dc383346173be45ce435254c2870199be93c305: Status 404 returned error can't find the container with id cef51de9f06232b2e7f69e719dc383346173be45ce435254c2870199be93c305 Nov 02 21:51:11.645880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.645855 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h"] Nov 02 21:51:11.645945 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:11.645891 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e68f91_e56f_41f1_a548_68bf84431876.slice/crio-a4defb87200c307837bf6d452d31715c800d791da9219f29e688f6018203e5c3 WatchSource:0}: Error finding container a4defb87200c307837bf6d452d31715c800d791da9219f29e688f6018203e5c3: Status 404 returned error can't find the container with id a4defb87200c307837bf6d452d31715c800d791da9219f29e688f6018203e5c3 Nov 02 21:51:11.646827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.646782 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.646891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.646853 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.646943 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.646887 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.648440 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.648414 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.648526 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.648465 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.648526 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:11.648490 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:12.627025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:12.626986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" event={"ID":"b9e68f91-e56f-41f1-a548-68bf84431876","Type":"ContainerStarted","Data":"2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d"} Nov 02 21:51:12.627209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:12.627040 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" event={"ID":"b9e68f91-e56f-41f1-a548-68bf84431876","Type":"ContainerStarted","Data":"a4defb87200c307837bf6d452d31715c800d791da9219f29e688f6018203e5c3"} Nov 02 21:51:12.629027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:12.628999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" event={"ID":"28033c5c-b32b-4d67-8326-bf25ca1afee4","Type":"ContainerStarted","Data":"b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834"} Nov 02 21:51:12.629186 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:12.629040 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" event={"ID":"28033c5c-b32b-4d67-8326-bf25ca1afee4","Type":"ContainerStarted","Data":"cef51de9f06232b2e7f69e719dc383346173be45ce435254c2870199be93c305"} Nov 02 21:51:12.850230 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:12.850182 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podStartSLOduration=2.850167195 podStartE2EDuration="2.850167195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:12.792899921 +0000 UTC m=+1019.954995041" watchObservedRunningTime="2026-11-02 21:51:12.850167195 +0000 UTC m=+1020.012262381" Nov 02 21:51:13.145666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.145599 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:13.147222 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.147189 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:51:13.147343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.147246 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 21:51:13.187025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.186991 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:13.189319 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.189269 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:51:13.189601 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.189554 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:51:13.516130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.515897 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podStartSLOduration=3.515875644 podStartE2EDuration="3.515875644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:12.920250436 +0000 UTC m=+1020.082345546" watchObservedRunningTime="2026-11-02 21:51:13.515875644 +0000 UTC m=+1020.677970766" Nov 02 21:51:13.521483 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.521285 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z"] Nov 02 21:51:13.533073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.533045 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.610691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.610115 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:51:13.611042 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.611020 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:51:13.622048 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.622003 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:51:13.622048 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.622005 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z"] Nov 02 21:51:13.622275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.622181 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:51:13.638336 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.638307 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.638508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.638474 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpq4\" (UniqueName: \"kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.638621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.638565 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.739500 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.739467 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpq4\" (UniqueName: \"kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.739648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.739545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.739648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.739635 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.740548 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.740520 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.742837 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.742115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.812740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.811419 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpq4\" (UniqueName: \"kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4\") pod \"maas-api-revoke-sub-e39527c9-zhj2z\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:13.852863 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:13.852814 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:14.146371 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.146332 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:51:14.146524 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.146395 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 21:51:14.193695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.193019 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:51:14.193695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.193083 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:51:14.311695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.311478 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr"] Nov 02 21:51:14.318706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.317663 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.345437 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.345273 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr"] Nov 02 21:51:14.345934 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.345905 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s857h\" (UniqueName: \"kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.346017 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.345962 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.346337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.346125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.416787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.414398 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z"] Nov 02 21:51:14.420627 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:14.420587 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b0cb4c_f277_46e6_8e82_22b1ccb6c006.slice/crio-988f77147131bc7321f7a2161998459bce6cb0aa5a97cf799b98d16795e4d581 WatchSource:0}: Error finding container 988f77147131bc7321f7a2161998459bce6cb0aa5a97cf799b98d16795e4d581: Status 404 returned error can't find the container with id 988f77147131bc7321f7a2161998459bce6cb0aa5a97cf799b98d16795e4d581 Nov 02 21:51:14.446806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.446784 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s857h\" (UniqueName: \"kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.446896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.446824 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.446896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.446862 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.447601 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.447578 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.448970 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.448951 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.515749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.515717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s857h\" (UniqueName: \"kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h\") pod \"maas-api-revoke-sub-c7d3c53f-z8wrr\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.632568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.632538 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:14.647396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:14.646611 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" event={"ID":"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006","Type":"ContainerStarted","Data":"988f77147131bc7321f7a2161998459bce6cb0aa5a97cf799b98d16795e4d581"} Nov 02 21:51:15.053678 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.053640 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr"] Nov 02 21:51:15.054801 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:15.054772 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74127a37_3725_40a7_99cd_73da4c359a28.slice/crio-b4a7109926fdb8111d9d64bf3a30b9edf7057c7bd3a5b92abdc621ac1a90770c WatchSource:0}: Error finding container b4a7109926fdb8111d9d64bf3a30b9edf7057c7bd3a5b92abdc621ac1a90770c: Status 404 returned error can't find the container with id b4a7109926fdb8111d9d64bf3a30b9edf7057c7bd3a5b92abdc621ac1a90770c Nov 02 21:51:15.145743 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.145717 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:51:15.145859 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.145765 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 21:51:15.188523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.188485 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:51:15.188817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.188784 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:51:15.705694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.703059 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" event={"ID":"74127a37-3725-40a7-99cd-73da4c359a28","Type":"ContainerStarted","Data":"8e83e5fdad603c16cc7cbe1ef0335b19a35b6edc21b02c3f4f738bd357181030"} Nov 02 21:51:15.705694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.703166 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" event={"ID":"74127a37-3725-40a7-99cd-73da4c359a28","Type":"ContainerStarted","Data":"b4a7109926fdb8111d9d64bf3a30b9edf7057c7bd3a5b92abdc621ac1a90770c"} Nov 02 21:51:15.715370 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.715332 5091 generic.go:356] "Generic (PLEG): container finished" podID="f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" containerID="483867d7a73992022072c8867a83fa5834514b47985b2e2984314ab9b202e939" exitCode=0 Nov 02 21:51:15.715544 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.715445 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" event={"ID":"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006","Type":"ContainerDied","Data":"483867d7a73992022072c8867a83fa5834514b47985b2e2984314ab9b202e939"} Nov 02 21:51:15.822537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:15.822486 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" podStartSLOduration=1.822472453 podStartE2EDuration="1.822472453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:15.817166034 +0000 UTC m=+1022.979261158" watchObservedRunningTime="2026-11-02 21:51:15.822472453 +0000 UTC m=+1022.984567572" Nov 02 21:51:16.145912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.145873 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:51:16.146307 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.145947 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 21:51:16.195237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.195199 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:51:16.195522 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.195493 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:51:16.502933 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.502845 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7kz5h-mjflh"] Nov 02 21:51:16.517804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.517770 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.552119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.552083 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7kz5h-mjflh"] Nov 02 21:51:16.713219 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.713187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.713498 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.713470 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qrml\" (UniqueName: \"kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.713606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.713529 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.713606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.713563 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.722844 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.722816 5091 generic.go:356] "Generic (PLEG): container finished" podID="74127a37-3725-40a7-99cd-73da4c359a28" containerID="8e83e5fdad603c16cc7cbe1ef0335b19a35b6edc21b02c3f4f738bd357181030" exitCode=0 Nov 02 21:51:16.723328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.723306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" event={"ID":"74127a37-3725-40a7-99cd-73da4c359a28","Type":"ContainerDied","Data":"8e83e5fdad603c16cc7cbe1ef0335b19a35b6edc21b02c3f4f738bd357181030"} Nov 02 21:51:16.817764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.817117 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrml\" (UniqueName: \"kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.817764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.817169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.817764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.817191 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.817764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.817220 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.818291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.818262 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.818374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.818314 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.820094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.820073 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.926450 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.926366 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrml\" (UniqueName: \"kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml\") pod \"periodic-gathering-7kz5h-mjflh\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:16.957632 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:16.957601 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:17.027027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.026997 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4\" (UniqueName: \"kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4\") pod \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " Nov 02 21:51:17.027222 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.027076 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca\") pod \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " Nov 02 21:51:17.027403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.027384 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca" pod "f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" (UID: "f6b0cb4c-f277-46e6-8e82-22b1ccb6c006"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:17.027476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.027431 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca\") pod \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\" (UID: \"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006\") " Nov 02 21:51:17.027972 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.027953 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:17.029211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.029176 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4" pod "f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" (UID: "f6b0cb4c-f277-46e6-8e82-22b1ccb6c006"). InnerVolumeSpecName "kube-api-access-2mpq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:17.029346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.029326 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca" pod "f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" (UID: "f6b0cb4c-f277-46e6-8e82-22b1ccb6c006"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:17.129610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.129577 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:17.129610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.129612 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mpq4\" (UniqueName: \"kubernetes.io/projected/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006-kube-api-access-2mpq4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:17.132792 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.132765 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:51:17.145925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.145892 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:51:17.146648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.146617 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 21:51:17.220068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.216348 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:51:17.220068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.216418 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:51:17.558110 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.558076 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7kz5h-mjflh"] Nov 02 21:51:17.558696 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:17.558650 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814cf912_1f61_41c1_89a8_4cf03427ea96.slice/crio-cd178b5a5b9dbc1beec2bd5db2e845f2be71382605be86028c39d9abcaf8991c WatchSource:0}: Error finding container cd178b5a5b9dbc1beec2bd5db2e845f2be71382605be86028c39d9abcaf8991c: Status 404 returned error can't find the container with id cd178b5a5b9dbc1beec2bd5db2e845f2be71382605be86028c39d9abcaf8991c Nov 02 21:51:17.733045 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.733008 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" event={"ID":"f6b0cb4c-f277-46e6-8e82-22b1ccb6c006","Type":"ContainerDied","Data":"988f77147131bc7321f7a2161998459bce6cb0aa5a97cf799b98d16795e4d581"} Nov 02 21:51:17.733197 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.733057 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="988f77147131bc7321f7a2161998459bce6cb0aa5a97cf799b98d16795e4d581" Nov 02 21:51:17.733197 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.733115 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z" Nov 02 21:51:17.737181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.737151 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" event={"ID":"814cf912-1f61-41c1-89a8-4cf03427ea96","Type":"ContainerStarted","Data":"75bc30cbc519942e051497c3833990cafa51ea4cd632fd015fbaee72d8582648"} Nov 02 21:51:17.737304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.737203 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" event={"ID":"814cf912-1f61-41c1-89a8-4cf03427ea96","Type":"ContainerStarted","Data":"cd178b5a5b9dbc1beec2bd5db2e845f2be71382605be86028c39d9abcaf8991c"} Nov 02 21:51:17.859886 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:17.859841 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" podStartSLOduration=1.8598271400000002 podStartE2EDuration="1.85982714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:17.857892779 +0000 UTC m=+1025.019987898" watchObservedRunningTime="2026-11-02 21:51:17.85982714 +0000 UTC m=+1025.021922260" Nov 02 21:51:18.013863 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.013827 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:18.038962 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.038928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca\") pod \"74127a37-3725-40a7-99cd-73da4c359a28\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " Nov 02 21:51:18.039151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.039058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca\") pod \"74127a37-3725-40a7-99cd-73da4c359a28\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " Nov 02 21:51:18.039151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.039118 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h\" (UniqueName: \"kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h\") pod \"74127a37-3725-40a7-99cd-73da4c359a28\" (UID: \"74127a37-3725-40a7-99cd-73da4c359a28\") " Nov 02 21:51:18.039381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.039356 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca" pod "74127a37-3725-40a7-99cd-73da4c359a28" (UID: "74127a37-3725-40a7-99cd-73da4c359a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:18.041091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.041064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca" pod "74127a37-3725-40a7-99cd-73da4c359a28" (UID: "74127a37-3725-40a7-99cd-73da4c359a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:18.041147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.041121 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h" pod "74127a37-3725-40a7-99cd-73da4c359a28" (UID: "74127a37-3725-40a7-99cd-73da4c359a28"). InnerVolumeSpecName "kube-api-access-s857h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:18.140310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.140269 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:18.140310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.140311 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s857h\" (UniqueName: \"kubernetes.io/projected/74127a37-3725-40a7-99cd-73da4c359a28-kube-api-access-s857h\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:18.140563 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.140329 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74127a37-3725-40a7-99cd-73da4c359a28-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:18.146090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.146059 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:51:18.146536 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.146115 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 21:51:18.185965 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.185934 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:51:18.186108 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.185978 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:51:18.750456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.750422 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" Nov 02 21:51:18.750736 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.750696 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr" event={"ID":"74127a37-3725-40a7-99cd-73da4c359a28","Type":"ContainerDied","Data":"b4a7109926fdb8111d9d64bf3a30b9edf7057c7bd3a5b92abdc621ac1a90770c"} Nov 02 21:51:18.750890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:18.750797 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a7109926fdb8111d9d64bf3a30b9edf7057c7bd3a5b92abdc621ac1a90770c" Nov 02 21:51:19.150189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:19.150159 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:19.150657 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:19.150632 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:19.151470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:19.151451 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 21:51:19.189833 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:19.189809 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:19.190101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:19.190080 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:19.190871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:19.190852 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 21:51:21.273576 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.273541 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r"] Nov 02 21:51:21.274088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.274073 5091 cpu_manager.go:414] "containerMap: removing container" podUID="74127a37-3725-40a7-99cd-73da4c359a28" containerName="revoke-keys" Nov 02 21:51:21.274137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.274089 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74127a37-3725-40a7-99cd-73da4c359a28" containerName="revoke-keys" Nov 02 21:51:21.274137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.274118 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" containerName="revoke-keys" Nov 02 21:51:21.274137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.274123 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" containerName="revoke-keys" Nov 02 21:51:21.274228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.274222 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" containerName="revoke-keys" Nov 02 21:51:21.274261 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.274234 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="74127a37-3725-40a7-99cd-73da4c359a28" containerName="revoke-keys" Nov 02 21:51:21.278810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.278790 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.334136 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.334112 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r"] Nov 02 21:51:21.376973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.376939 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmmh\" (UniqueName: \"kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.377113 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.376993 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.377151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.377114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.379559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.379535 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:51:21.379879 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.379861 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:51:21.379946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.379893 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:51:21.379946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.379943 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:51:21.478371 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.478342 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmmh\" (UniqueName: \"kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.478499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.478391 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.478499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.478454 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.479211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.479188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.480827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.480806 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.534921 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.534869 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmmh\" (UniqueName: \"kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh\") pod \"maas-api-revoke-sub-8f6fc593-pw87r\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.592111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.592086 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:21.964234 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:21.964201 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r"] Nov 02 21:51:21.964866 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:21.964842 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d255fd8_46f0_485c_b5a2_19d304bd4e6b.slice/crio-b6f863925dd9d01f87fcb155a80fe5b4c96bd2f89ffb0e6b199e34a96bbd0783 WatchSource:0}: Error finding container b6f863925dd9d01f87fcb155a80fe5b4c96bd2f89ffb0e6b199e34a96bbd0783: Status 404 returned error can't find the container with id b6f863925dd9d01f87fcb155a80fe5b4c96bd2f89ffb0e6b199e34a96bbd0783 Nov 02 21:51:22.781883 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:22.781848 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" event={"ID":"8d255fd8-46f0-485c-b5a2-19d304bd4e6b","Type":"ContainerStarted","Data":"e89abf3066653e40e15ae33da67432187b97652ad4b4a2eb68779b4929486f9d"} Nov 02 21:51:22.782261 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:22.781902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" event={"ID":"8d255fd8-46f0-485c-b5a2-19d304bd4e6b","Type":"ContainerStarted","Data":"b6f863925dd9d01f87fcb155a80fe5b4c96bd2f89ffb0e6b199e34a96bbd0783"} Nov 02 21:51:22.896522 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:22.896474 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" podStartSLOduration=1.896459656 podStartE2EDuration="1.896459656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:22.893342536 +0000 UTC m=+1030.055437659" watchObservedRunningTime="2026-11-02 21:51:22.896459656 +0000 UTC m=+1030.058554776" Nov 02 21:51:23.790238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:23.790145 5091 generic.go:356] "Generic (PLEG): container finished" podID="8d255fd8-46f0-485c-b5a2-19d304bd4e6b" containerID="e89abf3066653e40e15ae33da67432187b97652ad4b4a2eb68779b4929486f9d" exitCode=0 Nov 02 21:51:23.790238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:23.790213 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" event={"ID":"8d255fd8-46f0-485c-b5a2-19d304bd4e6b","Type":"ContainerDied","Data":"e89abf3066653e40e15ae33da67432187b97652ad4b4a2eb68779b4929486f9d"} Nov 02 21:51:24.231140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.231103 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w"] Nov 02 21:51:24.235115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.235097 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.291756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.291725 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w"] Nov 02 21:51:24.305622 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.305587 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.305796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.305769 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.305867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.305804 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntx5s\" (UniqueName: \"kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.406922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.406888 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.407094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.406993 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.407094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.407022 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntx5s\" (UniqueName: \"kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.407660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.407633 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.409187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.409161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.463439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.463411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntx5s\" (UniqueName: \"kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s\") pod \"maas-api-revoke-sub-83107559-4zs9w\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.548195 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.548133 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:24.920877 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:51:24.920848 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86be35_26f4_42dc_b485_e161cf95abba.slice/crio-630758c178c41b25c1fa0d7a806d2c623feae9b84cadcbeb99e9b1eccc1c6cba WatchSource:0}: Error finding container 630758c178c41b25c1fa0d7a806d2c623feae9b84cadcbeb99e9b1eccc1c6cba: Status 404 returned error can't find the container with id 630758c178c41b25c1fa0d7a806d2c623feae9b84cadcbeb99e9b1eccc1c6cba Nov 02 21:51:24.926086 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.926064 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w"] Nov 02 21:51:24.943321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:24.943296 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:25.014285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.014258 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca\") pod \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " Nov 02 21:51:25.014406 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.014360 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca\") pod \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " Nov 02 21:51:25.014406 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.014397 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh\" (UniqueName: \"kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh\") pod \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\" (UID: \"8d255fd8-46f0-485c-b5a2-19d304bd4e6b\") " Nov 02 21:51:25.014599 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.014579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca" pod "8d255fd8-46f0-485c-b5a2-19d304bd4e6b" (UID: "8d255fd8-46f0-485c-b5a2-19d304bd4e6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:25.014900 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.014875 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:25.016438 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.016418 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca" pod "8d255fd8-46f0-485c-b5a2-19d304bd4e6b" (UID: "8d255fd8-46f0-485c-b5a2-19d304bd4e6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:25.016531 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.016515 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh" pod "8d255fd8-46f0-485c-b5a2-19d304bd4e6b" (UID: "8d255fd8-46f0-485c-b5a2-19d304bd4e6b"). InnerVolumeSpecName "kube-api-access-8lmmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:25.116430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.116361 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:25.116430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.116388 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmmh\" (UniqueName: \"kubernetes.io/projected/8d255fd8-46f0-485c-b5a2-19d304bd4e6b-kube-api-access-8lmmh\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:25.820412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.820310 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" event={"ID":"9c86be35-26f4-42dc-b485-e161cf95abba","Type":"ContainerStarted","Data":"a563a26487d5fd0d082661bc2356cac14bbca3bdd831c322c00c21b3333a8656"} Nov 02 21:51:25.820412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.820403 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" event={"ID":"9c86be35-26f4-42dc-b485-e161cf95abba","Type":"ContainerStarted","Data":"630758c178c41b25c1fa0d7a806d2c623feae9b84cadcbeb99e9b1eccc1c6cba"} Nov 02 21:51:25.821823 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.821799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" event={"ID":"8d255fd8-46f0-485c-b5a2-19d304bd4e6b","Type":"ContainerDied","Data":"b6f863925dd9d01f87fcb155a80fe5b4c96bd2f89ffb0e6b199e34a96bbd0783"} Nov 02 21:51:25.821917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.821837 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f863925dd9d01f87fcb155a80fe5b4c96bd2f89ffb0e6b199e34a96bbd0783" Nov 02 21:51:25.821917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.821806 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r" Nov 02 21:51:25.974717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:25.974640 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" podStartSLOduration=1.9746261569999999 podStartE2EDuration="1.974626157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:25.97305666 +0000 UTC m=+1033.135151780" watchObservedRunningTime="2026-11-02 21:51:25.974626157 +0000 UTC m=+1033.136721277" Nov 02 21:51:26.832204 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:26.832165 5091 generic.go:356] "Generic (PLEG): container finished" podID="9c86be35-26f4-42dc-b485-e161cf95abba" containerID="a563a26487d5fd0d082661bc2356cac14bbca3bdd831c322c00c21b3333a8656" exitCode=0 Nov 02 21:51:26.832410 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:26.832245 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" event={"ID":"9c86be35-26f4-42dc-b485-e161cf95abba","Type":"ContainerDied","Data":"a563a26487d5fd0d082661bc2356cac14bbca3bdd831c322c00c21b3333a8656"} Nov 02 21:51:28.001873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.001851 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:28.045616 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.045590 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca\") pod \"9c86be35-26f4-42dc-b485-e161cf95abba\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " Nov 02 21:51:28.045768 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.045730 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s\" (UniqueName: \"kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s\") pod \"9c86be35-26f4-42dc-b485-e161cf95abba\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " Nov 02 21:51:28.045827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.045768 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca\") pod \"9c86be35-26f4-42dc-b485-e161cf95abba\" (UID: \"9c86be35-26f4-42dc-b485-e161cf95abba\") " Nov 02 21:51:28.046134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.046105 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca" pod "9c86be35-26f4-42dc-b485-e161cf95abba" (UID: "9c86be35-26f4-42dc-b485-e161cf95abba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:28.047716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.047689 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s" pod "9c86be35-26f4-42dc-b485-e161cf95abba" (UID: "9c86be35-26f4-42dc-b485-e161cf95abba"). InnerVolumeSpecName "kube-api-access-ntx5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:28.047868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.047854 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca" pod "9c86be35-26f4-42dc-b485-e161cf95abba" (UID: "9c86be35-26f4-42dc-b485-e161cf95abba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:28.147288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.147260 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntx5s\" (UniqueName: \"kubernetes.io/projected/9c86be35-26f4-42dc-b485-e161cf95abba-kube-api-access-ntx5s\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:28.147288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.147287 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:28.147478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.147298 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86be35-26f4-42dc-b485-e161cf95abba-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:51:28.847523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.847493 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" Nov 02 21:51:28.847713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.847486 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w" event={"ID":"9c86be35-26f4-42dc-b485-e161cf95abba","Type":"ContainerDied","Data":"630758c178c41b25c1fa0d7a806d2c623feae9b84cadcbeb99e9b1eccc1c6cba"} Nov 02 21:51:28.847713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:28.847621 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="630758c178c41b25c1fa0d7a806d2c623feae9b84cadcbeb99e9b1eccc1c6cba" Nov 02 21:51:37.113323 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:51:37.113283 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:51:43.910792 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.910760 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427"] Nov 02 21:51:43.911298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.911284 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9c86be35-26f4-42dc-b485-e161cf95abba" containerName="revoke-keys" Nov 02 21:51:43.911356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.911299 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c86be35-26f4-42dc-b485-e161cf95abba" containerName="revoke-keys" Nov 02 21:51:43.911356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.911330 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8d255fd8-46f0-485c-b5a2-19d304bd4e6b" containerName="revoke-keys" Nov 02 21:51:43.911356 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.911335 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d255fd8-46f0-485c-b5a2-19d304bd4e6b" containerName="revoke-keys" Nov 02 21:51:43.911455 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.911419 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c86be35-26f4-42dc-b485-e161cf95abba" containerName="revoke-keys" Nov 02 21:51:43.911455 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.911430 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d255fd8-46f0-485c-b5a2-19d304bd4e6b" containerName="revoke-keys" Nov 02 21:51:43.915960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.915942 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:43.973415 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:43.973378 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427"] Nov 02 21:51:44.017035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.016981 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.017035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017027 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zf9n\" (UniqueName: \"kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.017271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017111 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.017271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017148 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.017271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017155 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"kube-root-ca.crt\"" Nov 02 21:51:44.017430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017323 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"openshift-service-ca.crt\"" Nov 02 21:51:44.017430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017372 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.017430 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017381 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"default-dockercfg-xxmgz\"" Nov 02 21:51:44.017584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017497 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.017785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.017773 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"distinct-w5-96c052-kserve-self-signed-certs\"" Nov 02 21:51:44.118801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.118761 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.118801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.118814 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zf9n\" (UniqueName: \"kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.119081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.118876 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.119081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.118902 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.119198 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.119116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.119259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.119211 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.119443 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.119387 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.119966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.119811 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.120147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.120127 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.122322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.122298 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.122568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.122552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.175660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.175591 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zf9n\" (UniqueName: \"kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n\") pod \"distinct-w5-96c052-kserve-5c5c699d7-sz427\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.229308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.229272 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:44.615957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.615929 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427"] Nov 02 21:51:44.976870 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.976833 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" event={"ID":"6a1d46d9-4610-49b7-a708-7087610aff3b","Type":"ContainerStarted","Data":"571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a"} Nov 02 21:51:44.977358 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:44.976880 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" event={"ID":"6a1d46d9-4610-49b7-a708-7087610aff3b","Type":"ContainerStarted","Data":"de448575f5b8f19cb7721ca325fe1e5cb046c86f6205be0a64396e30f80e95d0"} Nov 02 21:51:45.092929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:45.092879 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podStartSLOduration=2.092865033 podStartE2EDuration="2.092865033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:45.087783483 +0000 UTC m=+1052.249878608" watchObservedRunningTime="2026-11-02 21:51:45.092865033 +0000 UTC m=+1052.254960153" Nov 02 21:51:45.230206 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:45.230129 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:54.235341 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:54.235296 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:54.254102 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:54.254073 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:51:55.074743 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:51:55.074714 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:52:01.008774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.008741 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp"] Nov 02 21:52:01.012620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.012603 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.116275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.116228 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"distinct-w6-fb8112-kserve-self-signed-certs\"" Nov 02 21:52:01.119001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.118978 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp"] Nov 02 21:52:01.125550 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.125518 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.125690 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.125582 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.125753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.125736 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnpt\" (UniqueName: \"kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.125793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.125768 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.125852 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.125839 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.125911 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.125899 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.226997 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.226959 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227072 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227212 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnpt\" (UniqueName: \"kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227684 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227643 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227697 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.227793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.227745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.231923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.231900 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.232328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.232303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.284076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.283994 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnpt\" (UniqueName: \"kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt\") pod \"distinct-w6-fb8112-kserve-7467697767-7mztp\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.325921 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.325891 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:01.736084 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:52:01.736057 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7896c66_d594_421a_8c95_91559ccbea35.slice/crio-0136ca39979bb242ab23af2e3ec17bfb8e2d7186d74d2ada41ee869be9f3b5e3 WatchSource:0}: Error finding container 0136ca39979bb242ab23af2e3ec17bfb8e2d7186d74d2ada41ee869be9f3b5e3: Status 404 returned error can't find the container with id 0136ca39979bb242ab23af2e3ec17bfb8e2d7186d74d2ada41ee869be9f3b5e3 Nov 02 21:52:01.738293 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:01.738263 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp"] Nov 02 21:52:02.116205 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:02.116121 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" event={"ID":"f7896c66-d594-421a-8c95-91559ccbea35","Type":"ContainerStarted","Data":"a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715"} Nov 02 21:52:02.116205 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:02.116169 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" event={"ID":"f7896c66-d594-421a-8c95-91559ccbea35","Type":"ContainerStarted","Data":"0136ca39979bb242ab23af2e3ec17bfb8e2d7186d74d2ada41ee869be9f3b5e3"} Nov 02 21:52:02.231093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:02.231045 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podStartSLOduration=2.231031243 podStartE2EDuration="2.231031243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:02.228921712 +0000 UTC m=+1069.391016835" watchObservedRunningTime="2026-11-02 21:52:02.231031243 +0000 UTC m=+1069.393126363" Nov 02 21:52:02.326826 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:02.326797 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:07.113447 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:52:07.113386 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:52:11.326397 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:11.326356 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:11.339248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:11.339220 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:12.197595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:12.197562 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:52:32.441770 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.441735 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm"] Nov 02 21:52:32.445850 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.445833 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.547933 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.547900 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"distinct-w2-9cc56b-kserve-self-signed-certs\"" Nov 02 21:52:32.551732 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.551709 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm"] Nov 02 21:52:32.562929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.562904 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.563042 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.562949 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpd8\" (UniqueName: \"kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.563115 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.563097 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.563170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.563148 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.563170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.563165 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.563260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.563207 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664269 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664348 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664414 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpd8\" (UniqueName: \"kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664852 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664757 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.664981 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.664957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.665057 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.665021 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.667287 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.667262 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.667414 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.667271 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.721638 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.721564 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpd8\" (UniqueName: \"kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8\") pod \"distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:32.758250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:32.758220 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:33.146987 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:33.146960 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm"] Nov 02 21:52:33.147506 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:52:33.147486 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb425e9fb_038d_4b47_859c_d5cf7c2c22dc.slice/crio-75ff6e3164d515bd1409b21035b2e5fcc2f6af52f386bc6d042ea169de1f381e WatchSource:0}: Error finding container 75ff6e3164d515bd1409b21035b2e5fcc2f6af52f386bc6d042ea169de1f381e: Status 404 returned error can't find the container with id 75ff6e3164d515bd1409b21035b2e5fcc2f6af52f386bc6d042ea169de1f381e Nov 02 21:52:33.344098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:33.344064 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" event={"ID":"b425e9fb-038d-4b47-859c-d5cf7c2c22dc","Type":"ContainerStarted","Data":"dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269"} Nov 02 21:52:33.344098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:33.344101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" event={"ID":"b425e9fb-038d-4b47-859c-d5cf7c2c22dc","Type":"ContainerStarted","Data":"75ff6e3164d515bd1409b21035b2e5fcc2f6af52f386bc6d042ea169de1f381e"} Nov 02 21:52:33.458931 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:33.458835 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podStartSLOduration=1.458821558 podStartE2EDuration="1.458821558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:33.456549794 +0000 UTC m=+1100.618644907" watchObservedRunningTime="2026-11-02 21:52:33.458821558 +0000 UTC m=+1100.620916688" Nov 02 21:52:33.759458 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:33.759388 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:37.114041 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:52:37.114002 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:52:42.758713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:42.758658 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:42.773184 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:42.773149 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:43.422324 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:43.422293 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:52:55.507393 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:55.507359 5091 generic.go:356] "Generic (PLEG): container finished" podID="814cf912-1f61-41c1-89a8-4cf03427ea96" containerID="75bc30cbc519942e051497c3833990cafa51ea4cd632fd015fbaee72d8582648" exitCode=0 Nov 02 21:52:55.507793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:55.507399 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" event={"ID":"814cf912-1f61-41c1-89a8-4cf03427ea96","Type":"ContainerDied","Data":"75bc30cbc519942e051497c3833990cafa51ea4cd632fd015fbaee72d8582648"} Nov 02 21:52:56.660496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.660472 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:52:56.715408 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.715375 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml\" (UniqueName: \"kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml\") pod \"814cf912-1f61-41c1-89a8-4cf03427ea96\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " Nov 02 21:52:56.715558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.715457 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle\") pod \"814cf912-1f61-41c1-89a8-4cf03427ea96\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " Nov 02 21:52:56.715558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.715505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path\") pod \"814cf912-1f61-41c1-89a8-4cf03427ea96\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " Nov 02 21:52:56.715702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.715633 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle\") pod \"814cf912-1f61-41c1-89a8-4cf03427ea96\" (UID: \"814cf912-1f61-41c1-89a8-4cf03427ea96\") " Nov 02 21:52:56.715954 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.715931 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path" pod "814cf912-1f61-41c1-89a8-4cf03427ea96" (UID: "814cf912-1f61-41c1-89a8-4cf03427ea96"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:56.716040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.715954 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle" pod "814cf912-1f61-41c1-89a8-4cf03427ea96" (UID: "814cf912-1f61-41c1-89a8-4cf03427ea96"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:56.716097 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.716073 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle" pod "814cf912-1f61-41c1-89a8-4cf03427ea96" (UID: "814cf912-1f61-41c1-89a8-4cf03427ea96"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:56.716279 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.716264 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-service-ca-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:52:56.716314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.716286 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/814cf912-1f61-41c1-89a8-4cf03427ea96-archives-path\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:52:56.716314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.716301 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/814cf912-1f61-41c1-89a8-4cf03427ea96-trusted-ca-bundle\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:52:56.717625 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.717606 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml" pod "814cf912-1f61-41c1-89a8-4cf03427ea96" (UID: "814cf912-1f61-41c1-89a8-4cf03427ea96"). InnerVolumeSpecName "kube-api-access-9qrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:56.817482 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:56.817387 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qrml\" (UniqueName: \"kubernetes.io/projected/814cf912-1f61-41c1-89a8-4cf03427ea96-kube-api-access-9qrml\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:52:57.522682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:57.522639 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" Nov 02 21:52:57.522851 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:57.522693 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7kz5h-mjflh" event={"ID":"814cf912-1f61-41c1-89a8-4cf03427ea96","Type":"ContainerDied","Data":"cd178b5a5b9dbc1beec2bd5db2e845f2be71382605be86028c39d9abcaf8991c"} Nov 02 21:52:57.522851 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:52:57.522740 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd178b5a5b9dbc1beec2bd5db2e845f2be71382605be86028c39d9abcaf8991c" Nov 02 21:53:07.116072 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:53:07.116032 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:53:11.402613 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.402576 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8"] Nov 02 21:53:11.403139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.403125 5091 cpu_manager.go:414] "containerMap: removing container" podUID="814cf912-1f61-41c1-89a8-4cf03427ea96" containerName="insights-gathering" Nov 02 21:53:11.403187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.403140 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814cf912-1f61-41c1-89a8-4cf03427ea96" containerName="insights-gathering" Nov 02 21:53:11.403256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.403246 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="814cf912-1f61-41c1-89a8-4cf03427ea96" containerName="insights-gathering" Nov 02 21:53:11.406879 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.406856 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.512449 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.512416 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8"] Nov 02 21:53:11.584135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584101 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584329 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584248 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584329 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584321 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584370 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584396 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqn5\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584428 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584557 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584531 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584554 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.584604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.584579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685200 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685111 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685200 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685170 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685200 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685194 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685223 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685254 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685357 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685403 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685542 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685597 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqn5\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685644 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685701 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685930 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.685973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.685956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.686069 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.686046 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.687772 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.687749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.687978 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.687961 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.741338 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.741308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:11.741468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:11.741341 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqn5\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:12.021429 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.021349 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:12.448594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.448195 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8"] Nov 02 21:53:12.454499 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.454461 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:12.454610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.454534 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:12.454610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.454564 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:12.634762 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.634719 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" event={"ID":"de1dd20c-2f12-4faf-9425-b2cf70938868","Type":"ContainerStarted","Data":"814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d"} Nov 02 21:53:12.634948 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.634780 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" event={"ID":"de1dd20c-2f12-4faf-9425-b2cf70938868","Type":"ContainerStarted","Data":"5908d55db302f73670dc05ede64ae1b557e15e60269dd07a89c00c2200473446"} Nov 02 21:53:12.752514 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:12.752392 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podStartSLOduration=1.752376619 podStartE2EDuration="1.752376619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:12.748363498 +0000 UTC m=+1139.910458619" watchObservedRunningTime="2026-11-02 21:53:12.752376619 +0000 UTC m=+1139.914471810" Nov 02 21:53:13.022572 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:13.022497 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:13.024203 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:13.024175 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:13.024348 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:13.024229 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:14.022467 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:14.022432 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:14.022979 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:14.022486 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:15.021661 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:15.021628 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:15.021866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:15.021710 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:16.021909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:16.021870 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:16.022304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:16.021929 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:17.021915 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:17.021879 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:17.022314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:17.021955 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:18.022420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:18.022384 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:18.022862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:18.022438 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:19.021836 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:19.021795 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:53:19.022024 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:19.021877 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 21:53:20.026363 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:20.026332 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:20.026794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:20.026771 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:20.027522 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:20.027502 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 21:53:26.883110 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.883074 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p"] Nov 02 21:53:26.887446 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.887427 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977292 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977254 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977300 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977391 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpk96\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977433 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977643 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977498 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977643 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977554 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977642 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977699 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.977787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.977770 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:26.987712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:26.987689 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p"] Nov 02 21:53:27.078284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078257 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpk96\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078295 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078323 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078359 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078408 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078436 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078490 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078848 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078940 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078893 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.078940 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.078904 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.079162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.079140 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.079333 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.079307 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.079528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.079505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.079573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.079529 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.080908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.080885 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.081239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.081219 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.134925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.134859 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.135164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.135139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpk96\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.201143 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.201111 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:27.497566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.497494 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb"] Nov 02 21:53:27.502724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.502709 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.555270 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.555233 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ffe386-openshift-default-dockercfg-8xxrb\"" Nov 02 21:53:27.559943 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.559922 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb"] Nov 02 21:53:27.583538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583517 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583598 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6626\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583638 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583750 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583658 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583750 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583692 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583872 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583945 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583927 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.583986 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.584022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.583995 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.629891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.629863 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p"] Nov 02 21:53:27.630003 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:53:27.629936 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857f1f53_f73a_4026_9c6c_5a9013252e88.slice/crio-7da5ec6302f8c96f06bfeb98991cfd8ffe815577e2d104bbd40816a5b15edefc WatchSource:0}: Error finding container 7da5ec6302f8c96f06bfeb98991cfd8ffe815577e2d104bbd40816a5b15edefc: Status 404 returned error can't find the container with id 7da5ec6302f8c96f06bfeb98991cfd8ffe815577e2d104bbd40816a5b15edefc Nov 02 21:53:27.632170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.632142 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:27.632238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.632200 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:27.632238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.632226 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:27.685401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685376 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685492 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6626\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685492 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685487 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685505 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685526 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685583 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685617 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.685804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685691 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.686006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.685875 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.686100 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.686081 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.686167 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.686148 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.686328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.686301 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.686398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.686336 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.688165 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.688145 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.688383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.688364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.741545 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.741514 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6626\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.741709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.741619 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.758145 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.758062 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" event={"ID":"857f1f53-f73a-4026-9c6c-5a9013252e88","Type":"ContainerStarted","Data":"bee3a8782acff51770d8322b14332e2ee0ecf1a3969873d3c527ed686f17e471"} Nov 02 21:53:27.758145 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.758138 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" event={"ID":"857f1f53-f73a-4026-9c6c-5a9013252e88","Type":"ContainerStarted","Data":"7da5ec6302f8c96f06bfeb98991cfd8ffe815577e2d104bbd40816a5b15edefc"} Nov 02 21:53:27.816846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.816814 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:27.876922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:27.876868 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podStartSLOduration=1.876852832 podStartE2EDuration="1.876852832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:27.873250728 +0000 UTC m=+1155.035345850" watchObservedRunningTime="2026-11-02 21:53:27.876852832 +0000 UTC m=+1155.038947952" Nov 02 21:53:28.202257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.202231 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:28.203853 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.203828 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:28.203982 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.203886 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:28.211233 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:53:28.211205 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf534c2d6_402a_45e4_8b5a_56e91520bdfc.slice/crio-45aedd96597d91da8b984bb44dacb554455b63969ff70675e6f5809550df267d WatchSource:0}: Error finding container 45aedd96597d91da8b984bb44dacb554455b63969ff70675e6f5809550df267d: Status 404 returned error can't find the container with id 45aedd96597d91da8b984bb44dacb554455b63969ff70675e6f5809550df267d Nov 02 21:53:28.212000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.211976 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb"] Nov 02 21:53:28.214077 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.214043 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:28.214192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.214121 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:28.214192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.214167 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:28.772693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.770485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" event={"ID":"f534c2d6-402a-45e4-8b5a-56e91520bdfc","Type":"ContainerStarted","Data":"8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8"} Nov 02 21:53:28.772693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.770550 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" event={"ID":"f534c2d6-402a-45e4-8b5a-56e91520bdfc","Type":"ContainerStarted","Data":"45aedd96597d91da8b984bb44dacb554455b63969ff70675e6f5809550df267d"} Nov 02 21:53:28.817702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.817636 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:28.897833 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:28.897778 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podStartSLOduration=1.897762545 podStartE2EDuration="1.897762545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:28.895170478 +0000 UTC m=+1156.057265638" watchObservedRunningTime="2026-11-02 21:53:28.897762545 +0000 UTC m=+1156.059857718" Nov 02 21:53:29.202640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:29.202185 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:29.202640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:29.202245 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:29.818629 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:29.818592 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:53:29.818841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:29.818645 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 21:53:30.213159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:30.213097 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:30.213729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:30.213161 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:30.817703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:30.817474 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:53:30.817703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:30.817540 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 21:53:31.204696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:31.204076 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:31.204696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:31.204140 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:31.818694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:31.818040 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:53:31.818694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:31.818106 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 21:53:32.206698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:32.206061 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:32.206698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:32.206124 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:32.818365 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:32.818329 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:53:32.818577 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:32.818396 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 21:53:33.202695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:33.202104 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:33.202695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:33.202164 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:33.817550 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:33.817512 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:53:33.817730 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:33.817564 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 21:53:34.201559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:34.201520 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 21:53:34.201753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:34.201578 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 21:53:34.817886 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:34.817847 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:53:34.818391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:34.817912 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 21:53:35.206686 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:35.206642 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:35.207060 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:35.207038 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:35.207868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:35.207848 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 21:53:35.823502 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:35.823465 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:35.824024 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:35.823999 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:35.825233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:35.825212 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 21:53:37.112404 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:53:37.112372 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:53:40.026531 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.026498 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl"] Nov 02 21:53:40.031073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.031049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.113892 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.113853 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl"] Nov 02 21:53:40.131074 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.131036 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:53:40.131245 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.131074 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:53:40.131245 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.131073 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:53:40.132602 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.132579 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:53:40.159033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.159003 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.159168 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.159046 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrxd\" (UniqueName: \"kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.159259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.159240 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.260405 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.260373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrxd\" (UniqueName: \"kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.260582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.260508 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.260582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.260571 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.261237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.261207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.262985 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.262955 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.317863 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.317785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrxd\" (UniqueName: \"kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd\") pod \"maas-api-revoke-sub-7d227f2f-mszvl\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.344871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.344838 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:40.736875 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:53:40.736843 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00af4ae4_38f5_47b9_b914_3f11179abb86.slice/crio-17adb3dc4e04a4f3dde3b65a59e4719ca4e0e4fbb0df8d10610f1f8f5748d961 WatchSource:0}: Error finding container 17adb3dc4e04a4f3dde3b65a59e4719ca4e0e4fbb0df8d10610f1f8f5748d961: Status 404 returned error can't find the container with id 17adb3dc4e04a4f3dde3b65a59e4719ca4e0e4fbb0df8d10610f1f8f5748d961 Nov 02 21:53:40.737841 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.737814 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl"] Nov 02 21:53:40.896876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:40.896844 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" event={"ID":"00af4ae4-38f5-47b9-b914-3f11179abb86","Type":"ContainerStarted","Data":"17adb3dc4e04a4f3dde3b65a59e4719ca4e0e4fbb0df8d10610f1f8f5748d961"} Nov 02 21:53:41.905049 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:41.904957 5091 generic.go:356] "Generic (PLEG): container finished" podID="00af4ae4-38f5-47b9-b914-3f11179abb86" containerID="aff7d5e9adbdc0c20543ed6ca12513c014fac3b4ed9f86e6ce7730e619538704" exitCode=0 Nov 02 21:53:41.905435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:41.905041 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" event={"ID":"00af4ae4-38f5-47b9-b914-3f11179abb86","Type":"ContainerDied","Data":"aff7d5e9adbdc0c20543ed6ca12513c014fac3b4ed9f86e6ce7730e619538704"} Nov 02 21:53:42.331998 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.331929 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj"] Nov 02 21:53:42.336073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.336053 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.484995 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.484962 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485074 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485217 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485271 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485313 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485449 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485417 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6cw\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485506 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485484 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.485557 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.485519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.542647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.542621 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj"] Nov 02 21:53:42.587121 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587039 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587121 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587078 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587125 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587155 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587188 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6cw\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587588 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587354 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587588 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587426 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587604 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587633 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587724 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587705 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.587912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.587753 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.588187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.588165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.589712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.589688 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.589813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.589714 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.643279 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.643246 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.643394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.643348 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6cw\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:42.649318 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:42.649302 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:43.057275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.057252 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:43.081462 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:53:43.081419 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0c8f89_1afe_41ad_85cf_6fbdb9c7ecbd.slice/crio-ac851167bc85670d7064754571bd8a4ebefa3a46403e5649289eb873b2cf878f WatchSource:0}: Error finding container ac851167bc85670d7064754571bd8a4ebefa3a46403e5649289eb873b2cf878f: Status 404 returned error can't find the container with id ac851167bc85670d7064754571bd8a4ebefa3a46403e5649289eb873b2cf878f Nov 02 21:53:43.082296 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.082267 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj"] Nov 02 21:53:43.084789 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.084744 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:43.084884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.084829 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:43.084884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.084873 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:43.091969 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.091946 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca\") pod \"00af4ae4-38f5-47b9-b914-3f11179abb86\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " Nov 02 21:53:43.092061 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.091992 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca\") pod \"00af4ae4-38f5-47b9-b914-3f11179abb86\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " Nov 02 21:53:43.092360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.092327 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca" pod "00af4ae4-38f5-47b9-b914-3f11179abb86" (UID: "00af4ae4-38f5-47b9-b914-3f11179abb86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:53:43.094247 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.094222 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca" pod "00af4ae4-38f5-47b9-b914-3f11179abb86" (UID: "00af4ae4-38f5-47b9-b914-3f11179abb86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:53:43.193255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.193165 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd\" (UniqueName: \"kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd\") pod \"00af4ae4-38f5-47b9-b914-3f11179abb86\" (UID: \"00af4ae4-38f5-47b9-b914-3f11179abb86\") " Nov 02 21:53:43.193728 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.193625 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:53:43.193728 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.193650 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af4ae4-38f5-47b9-b914-3f11179abb86-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:53:43.196496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.196467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd" pod "00af4ae4-38f5-47b9-b914-3f11179abb86" (UID: "00af4ae4-38f5-47b9-b914-3f11179abb86"). InnerVolumeSpecName "kube-api-access-rjrxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:53:43.294352 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.294320 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjrxd\" (UniqueName: \"kubernetes.io/projected/00af4ae4-38f5-47b9-b914-3f11179abb86-kube-api-access-rjrxd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:53:43.923301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.923260 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" event={"ID":"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd","Type":"ContainerStarted","Data":"442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb"} Nov 02 21:53:43.923497 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.923346 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" event={"ID":"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd","Type":"ContainerStarted","Data":"ac851167bc85670d7064754571bd8a4ebefa3a46403e5649289eb873b2cf878f"} Nov 02 21:53:43.924717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.924692 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" event={"ID":"00af4ae4-38f5-47b9-b914-3f11179abb86","Type":"ContainerDied","Data":"17adb3dc4e04a4f3dde3b65a59e4719ca4e0e4fbb0df8d10610f1f8f5748d961"} Nov 02 21:53:43.924827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.924719 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl" Nov 02 21:53:43.924827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:43.924731 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17adb3dc4e04a4f3dde3b65a59e4719ca4e0e4fbb0df8d10610f1f8f5748d961" Nov 02 21:53:44.143920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:44.143866 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podStartSLOduration=2.143852966 podStartE2EDuration="2.143852966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:44.139721911 +0000 UTC m=+1171.301817030" watchObservedRunningTime="2026-11-02 21:53:44.143852966 +0000 UTC m=+1171.305948085" Nov 02 21:53:44.650182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:44.650148 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:44.651869 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:44.651837 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:53:44.651997 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:44.651886 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:53:45.649956 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:45.649925 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:53:45.650322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:45.649980 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:53:46.650159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:46.650126 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:53:46.650624 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:46.650187 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:53:47.650085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:47.650049 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:53:47.650264 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:47.650104 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:53:48.649761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:48.649725 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:53:48.649936 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:48.649790 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:53:49.650272 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:49.650237 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:53:49.650790 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:49.650299 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:53:50.654473 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:50.654444 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:50.654896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:50.654877 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:50.655633 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:50.655612 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 21:53:58.538181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.538144 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w"] Nov 02 21:53:58.539099 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.539076 5091 cpu_manager.go:414] "containerMap: removing container" podUID="00af4ae4-38f5-47b9-b914-3f11179abb86" containerName="revoke-keys" Nov 02 21:53:58.539158 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.539101 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00af4ae4-38f5-47b9-b914-3f11179abb86" containerName="revoke-keys" Nov 02 21:53:58.539306 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.539293 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="00af4ae4-38f5-47b9-b914-3f11179abb86" containerName="revoke-keys" Nov 02 21:53:58.544813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.544797 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.596211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.596186 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w"] Nov 02 21:53:58.640648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.640615 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:53:58.640648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.640625 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:53:58.640850 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.640623 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:53:58.640850 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.640741 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:53:58.678992 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.678969 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.679106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.679021 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.679244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.679226 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77bd9\" (UniqueName: \"kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.780427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.780400 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77bd9\" (UniqueName: \"kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.780570 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.780515 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.780570 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.780557 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.781174 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.781151 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.782959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.782931 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.836628 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.836570 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77bd9\" (UniqueName: \"kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9\") pod \"maas-api-revoke-sub-b1035b95-ctj9w\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:58.858532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:58.858512 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:53:59.239563 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:53:59.239525 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfedef5dc_cec8_498d_b928_862f7a0ef23d.slice/crio-fea6de13d00f46eb8571d39b51f215fb38dbd8790831c6c0534a4f5ad2b77f81 WatchSource:0}: Error finding container fea6de13d00f46eb8571d39b51f215fb38dbd8790831c6c0534a4f5ad2b77f81: Status 404 returned error can't find the container with id fea6de13d00f46eb8571d39b51f215fb38dbd8790831c6c0534a4f5ad2b77f81 Nov 02 21:53:59.240067 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:53:59.240045 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w"] Nov 02 21:54:00.049967 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:00.049936 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" event={"ID":"fedef5dc-cec8-498d-b928-862f7a0ef23d","Type":"ContainerStarted","Data":"90f2262b0a230a416ff4eb7bdd4254169e2b45080f3cb7ab7549fdf9248457b2"} Nov 02 21:54:00.050343 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:00.049987 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" event={"ID":"fedef5dc-cec8-498d-b928-862f7a0ef23d","Type":"ContainerStarted","Data":"fea6de13d00f46eb8571d39b51f215fb38dbd8790831c6c0534a4f5ad2b77f81"} Nov 02 21:54:00.164905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:00.164862 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" podStartSLOduration=2.164849165 podStartE2EDuration="2.164849165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:00.161118669 +0000 UTC m=+1187.323213791" watchObservedRunningTime="2026-11-02 21:54:00.164849165 +0000 UTC m=+1187.326944314" Nov 02 21:54:01.058866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:01.058835 5091 generic.go:356] "Generic (PLEG): container finished" podID="fedef5dc-cec8-498d-b928-862f7a0ef23d" containerID="90f2262b0a230a416ff4eb7bdd4254169e2b45080f3cb7ab7549fdf9248457b2" exitCode=0 Nov 02 21:54:01.059274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:01.058922 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" event={"ID":"fedef5dc-cec8-498d-b928-862f7a0ef23d","Type":"ContainerDied","Data":"90f2262b0a230a416ff4eb7bdd4254169e2b45080f3cb7ab7549fdf9248457b2"} Nov 02 21:54:02.210986 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.210960 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:54:02.319610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.319535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca\") pod \"fedef5dc-cec8-498d-b928-862f7a0ef23d\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " Nov 02 21:54:02.319810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.319791 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca\") pod \"fedef5dc-cec8-498d-b928-862f7a0ef23d\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " Nov 02 21:54:02.319881 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.319832 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9\" (UniqueName: \"kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9\") pod \"fedef5dc-cec8-498d-b928-862f7a0ef23d\" (UID: \"fedef5dc-cec8-498d-b928-862f7a0ef23d\") " Nov 02 21:54:02.320178 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.320152 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca" pod "fedef5dc-cec8-498d-b928-862f7a0ef23d" (UID: "fedef5dc-cec8-498d-b928-862f7a0ef23d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:02.321460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.321430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca" pod "fedef5dc-cec8-498d-b928-862f7a0ef23d" (UID: "fedef5dc-cec8-498d-b928-862f7a0ef23d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:02.321649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.321630 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9" pod "fedef5dc-cec8-498d-b928-862f7a0ef23d" (UID: "fedef5dc-cec8-498d-b928-862f7a0ef23d"). InnerVolumeSpecName "kube-api-access-77bd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:02.422377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.422347 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:02.422377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.422374 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77bd9\" (UniqueName: \"kubernetes.io/projected/fedef5dc-cec8-498d-b928-862f7a0ef23d-kube-api-access-77bd9\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:02.422377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:02.422384 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fedef5dc-cec8-498d-b928-862f7a0ef23d-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:03.074368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:03.074338 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" Nov 02 21:54:03.074528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:03.074342 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w" event={"ID":"fedef5dc-cec8-498d-b928-862f7a0ef23d","Type":"ContainerDied","Data":"fea6de13d00f46eb8571d39b51f215fb38dbd8790831c6c0534a4f5ad2b77f81"} Nov 02 21:54:03.074528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:03.074476 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fea6de13d00f46eb8571d39b51f215fb38dbd8790831c6c0534a4f5ad2b77f81" Nov 02 21:54:07.114201 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:54:07.114164 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:54:11.418124 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.418090 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb"] Nov 02 21:54:11.418657 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.418642 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fedef5dc-cec8-498d-b928-862f7a0ef23d" containerName="revoke-keys" Nov 02 21:54:11.418745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.418659 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fedef5dc-cec8-498d-b928-862f7a0ef23d" containerName="revoke-keys" Nov 02 21:54:11.418812 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.418802 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fedef5dc-cec8-498d-b928-862f7a0ef23d" containerName="revoke-keys" Nov 02 21:54:11.422584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.422568 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.479409 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.479381 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb"] Nov 02 21:54:11.522616 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.522588 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.522768 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.522742 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.522858 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.522841 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h476\" (UniqueName: \"kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.523961 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.523921 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:11.524139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.524124 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:11.524200 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.524142 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:54:11.524815 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.524797 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:11.624242 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.624213 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.624374 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.624359 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.624439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.624422 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h476\" (UniqueName: \"kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.624920 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.624895 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.626761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.626745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.681140 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.681080 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h476\" (UniqueName: \"kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476\") pod \"maas-api-revoke-sub-10dc4518-p6klb\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:11.736041 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:11.736012 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:12.111534 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:12.111497 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefddd336_a95e_4fea_be2c_a2bcb390bbec.slice/crio-4d5b69f015f6b47c09ddf7f2615e9f20491b727be73ee390499681234b61c2bf WatchSource:0}: Error finding container 4d5b69f015f6b47c09ddf7f2615e9f20491b727be73ee390499681234b61c2bf: Status 404 returned error can't find the container with id 4d5b69f015f6b47c09ddf7f2615e9f20491b727be73ee390499681234b61c2bf Nov 02 21:54:12.112382 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:12.112339 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb"] Nov 02 21:54:12.145047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:12.145022 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" event={"ID":"efddd336-a95e-4fea-be2c-a2bcb390bbec","Type":"ContainerStarted","Data":"4d5b69f015f6b47c09ddf7f2615e9f20491b727be73ee390499681234b61c2bf"} Nov 02 21:54:13.155556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:13.155510 5091 generic.go:356] "Generic (PLEG): container finished" podID="efddd336-a95e-4fea-be2c-a2bcb390bbec" containerID="8d9795252907ab6c099582f9eb39ae53deed0375883d1e8cb54ed23871bf362b" exitCode=0 Nov 02 21:54:13.156013 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:13.155587 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" event={"ID":"efddd336-a95e-4fea-be2c-a2bcb390bbec","Type":"ContainerDied","Data":"8d9795252907ab6c099582f9eb39ae53deed0375883d1e8cb54ed23871bf362b"} Nov 02 21:54:13.161171 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:13.161149 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:54:13.163437 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:13.163414 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:54:14.308129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.308105 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:14.354049 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.354025 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476\" (UniqueName: \"kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476\") pod \"efddd336-a95e-4fea-be2c-a2bcb390bbec\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " Nov 02 21:54:14.354172 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.354158 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca\") pod \"efddd336-a95e-4fea-be2c-a2bcb390bbec\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " Nov 02 21:54:14.354221 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.354207 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca\") pod \"efddd336-a95e-4fea-be2c-a2bcb390bbec\" (UID: \"efddd336-a95e-4fea-be2c-a2bcb390bbec\") " Nov 02 21:54:14.354563 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.354540 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca" pod "efddd336-a95e-4fea-be2c-a2bcb390bbec" (UID: "efddd336-a95e-4fea-be2c-a2bcb390bbec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:14.355976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.355956 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476" pod "efddd336-a95e-4fea-be2c-a2bcb390bbec" (UID: "efddd336-a95e-4fea-be2c-a2bcb390bbec"). InnerVolumeSpecName "kube-api-access-8h476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:14.356074 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.355991 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca" pod "efddd336-a95e-4fea-be2c-a2bcb390bbec" (UID: "efddd336-a95e-4fea-be2c-a2bcb390bbec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:14.455421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.455348 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:14.455421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.455381 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efddd336-a95e-4fea-be2c-a2bcb390bbec-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:14.455421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:14.455397 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h476\" (UniqueName: \"kubernetes.io/projected/efddd336-a95e-4fea-be2c-a2bcb390bbec-kube-api-access-8h476\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:15.172053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:15.172019 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" event={"ID":"efddd336-a95e-4fea-be2c-a2bcb390bbec","Type":"ContainerDied","Data":"4d5b69f015f6b47c09ddf7f2615e9f20491b727be73ee390499681234b61c2bf"} Nov 02 21:54:15.172214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:15.172069 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d5b69f015f6b47c09ddf7f2615e9f20491b727be73ee390499681234b61c2bf" Nov 02 21:54:15.172214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:15.172121 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb" Nov 02 21:54:16.289512 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.289476 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw"] Nov 02 21:54:16.290059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.290044 5091 cpu_manager.go:414] "containerMap: removing container" podUID="efddd336-a95e-4fea-be2c-a2bcb390bbec" containerName="revoke-keys" Nov 02 21:54:16.290107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.290060 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efddd336-a95e-4fea-be2c-a2bcb390bbec" containerName="revoke-keys" Nov 02 21:54:16.290176 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.290166 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="efddd336-a95e-4fea-be2c-a2bcb390bbec" containerName="revoke-keys" Nov 02 21:54:16.293829 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.293801 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.327285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.327251 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh"] Nov 02 21:54:16.331808 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.331785 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.372314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.372281 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"distinct-2-w3-64438f-kserve-self-signed-certs\"" Nov 02 21:54:16.375690 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.375652 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.375807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.375757 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.375923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.375906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.375982 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.375950 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.376050 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.376033 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.376089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.376057 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.376089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.376073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.376189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.376097 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvcb\" (UniqueName: \"kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.376275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.376260 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9f5x\" (UniqueName: \"kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.376420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.376397 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw"] Nov 02 21:54:16.383767 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.383752 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:16.395317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.395292 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:16.402239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.402214 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh"] Nov 02 21:54:16.446755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.446724 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:16.446984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.446846 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:54:16.484143 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.484106 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9f5x\" (UniqueName: \"kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.484445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.484419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.484604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.484585 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.486662 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.485745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.486807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.486698 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.486807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.486751 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.486807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.486788 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.487070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.486815 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.487070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.486845 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.487070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.486889 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvcb\" (UniqueName: \"kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.487426 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.487143 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.487426 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.487402 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.503731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.488010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.503731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.495093 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.503731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.495361 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.503731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.498327 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.540230 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.540155 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9f5x\" (UniqueName: \"kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x\") pod \"maas-api-revoke-sub-3641a89f-69dnh\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:16.542849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.542793 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvcb\" (UniqueName: \"kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb\") pod \"distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.608784 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.608753 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:16.646275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:16.646243 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:17.004063 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:17.004033 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92a70dac_c289_41e3_94f2_e522b78fd6c9.slice/crio-c12a8e4fd1435849d08e9d7e9f55953008a38c3dad136a7949834eb60f3fc234 WatchSource:0}: Error finding container c12a8e4fd1435849d08e9d7e9f55953008a38c3dad136a7949834eb60f3fc234: Status 404 returned error can't find the container with id c12a8e4fd1435849d08e9d7e9f55953008a38c3dad136a7949834eb60f3fc234 Nov 02 21:54:17.004885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.004862 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw"] Nov 02 21:54:17.053703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.053680 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh"] Nov 02 21:54:17.053824 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:17.053796 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfbc9c2_d9ba_4951_ac7b_7d4010549361.slice/crio-61ea631de4808aae6678f2474d4c655fdce7fd7655270dc272060b62ffa3a289 WatchSource:0}: Error finding container 61ea631de4808aae6678f2474d4c655fdce7fd7655270dc272060b62ffa3a289: Status 404 returned error can't find the container with id 61ea631de4808aae6678f2474d4c655fdce7fd7655270dc272060b62ffa3a289 Nov 02 21:54:17.188511 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.188478 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" event={"ID":"3bfbc9c2-d9ba-4951-ac7b-7d4010549361","Type":"ContainerStarted","Data":"61ea631de4808aae6678f2474d4c655fdce7fd7655270dc272060b62ffa3a289"} Nov 02 21:54:17.190250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.190225 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" event={"ID":"92a70dac-c289-41e3-94f2-e522b78fd6c9","Type":"ContainerStarted","Data":"69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007"} Nov 02 21:54:17.190386 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.190256 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" event={"ID":"92a70dac-c289-41e3-94f2-e522b78fd6c9","Type":"ContainerStarted","Data":"c12a8e4fd1435849d08e9d7e9f55953008a38c3dad136a7949834eb60f3fc234"} Nov 02 21:54:17.306885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.306843 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podStartSLOduration=1.306830546 podStartE2EDuration="1.306830546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:17.302791341 +0000 UTC m=+1204.464886460" watchObservedRunningTime="2026-11-02 21:54:17.306830546 +0000 UTC m=+1204.468925666" Nov 02 21:54:17.609838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:17.609750 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:18.199576 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:18.199493 5091 generic.go:356] "Generic (PLEG): container finished" podID="3bfbc9c2-d9ba-4951-ac7b-7d4010549361" containerID="715b474e3f64a7f3b0eab7f12bee7e4d26e8102ceb52e89063fd3e7bc9dbbabb" exitCode=0 Nov 02 21:54:18.199747 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:18.199577 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" event={"ID":"3bfbc9c2-d9ba-4951-ac7b-7d4010549361","Type":"ContainerDied","Data":"715b474e3f64a7f3b0eab7f12bee7e4d26e8102ceb52e89063fd3e7bc9dbbabb"} Nov 02 21:54:19.349582 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.349551 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:19.438361 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.438333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca\") pod \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " Nov 02 21:54:19.438519 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.438457 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca\") pod \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " Nov 02 21:54:19.438519 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.438514 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x\" (UniqueName: \"kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x\") pod \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\" (UID: \"3bfbc9c2-d9ba-4951-ac7b-7d4010549361\") " Nov 02 21:54:19.438686 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.438649 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca" pod "3bfbc9c2-d9ba-4951-ac7b-7d4010549361" (UID: "3bfbc9c2-d9ba-4951-ac7b-7d4010549361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:19.438846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.438830 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:19.440588 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.440558 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca" pod "3bfbc9c2-d9ba-4951-ac7b-7d4010549361" (UID: "3bfbc9c2-d9ba-4951-ac7b-7d4010549361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:19.440715 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.440621 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x" pod "3bfbc9c2-d9ba-4951-ac7b-7d4010549361" (UID: "3bfbc9c2-d9ba-4951-ac7b-7d4010549361"). InnerVolumeSpecName "kube-api-access-d9f5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:19.540328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.540259 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:19.540328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:19.540284 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9f5x\" (UniqueName: \"kubernetes.io/projected/3bfbc9c2-d9ba-4951-ac7b-7d4010549361-kube-api-access-d9f5x\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:20.215363 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:20.215330 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" Nov 02 21:54:20.215522 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:20.215361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh" event={"ID":"3bfbc9c2-d9ba-4951-ac7b-7d4010549361","Type":"ContainerDied","Data":"61ea631de4808aae6678f2474d4c655fdce7fd7655270dc272060b62ffa3a289"} Nov 02 21:54:20.215522 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:20.215411 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ea631de4808aae6678f2474d4c655fdce7fd7655270dc272060b62ffa3a289" Nov 02 21:54:26.609755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:26.609661 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:26.623595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:26.623570 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:27.291896 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.291864 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:54:27.459986 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.459952 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk"] Nov 02 21:54:27.460988 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.460967 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3bfbc9c2-d9ba-4951-ac7b-7d4010549361" containerName="revoke-keys" Nov 02 21:54:27.461098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.460989 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfbc9c2-d9ba-4951-ac7b-7d4010549361" containerName="revoke-keys" Nov 02 21:54:27.461151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.461139 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfbc9c2-d9ba-4951-ac7b-7d4010549361" containerName="revoke-keys" Nov 02 21:54:27.465370 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.465346 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.518640 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.518608 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:27.518875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.518836 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:27.519612 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.519060 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:54:27.519612 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.519064 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:27.525794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.525760 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.525932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.525859 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27js\" (UniqueName: \"kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.525932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.525921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.526641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.526620 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk"] Nov 02 21:54:27.627126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.627092 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27js\" (UniqueName: \"kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.627571 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.627151 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.627571 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.627248 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.627867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.627841 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.629596 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.629573 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.683322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.683293 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27js\" (UniqueName: \"kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js\") pod \"maas-api-revoke-sub-f20e6269-rqvqk\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:27.781185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:27.781152 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:28.157840 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:28.157807 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk"] Nov 02 21:54:28.157949 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:28.157818 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187edd17_e602_4707_8a36_7ee14162b6b9.slice/crio-c0654ade2402f076bd158e3900a4f700069e4e00c157c3b69e446d37b50cf24b WatchSource:0}: Error finding container c0654ade2402f076bd158e3900a4f700069e4e00c157c3b69e446d37b50cf24b: Status 404 returned error can't find the container with id c0654ade2402f076bd158e3900a4f700069e4e00c157c3b69e446d37b50cf24b Nov 02 21:54:28.289424 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:28.289388 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" event={"ID":"187edd17-e602-4707-8a36-7ee14162b6b9","Type":"ContainerStarted","Data":"c0654ade2402f076bd158e3900a4f700069e4e00c157c3b69e446d37b50cf24b"} Nov 02 21:54:29.299831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:29.299751 5091 generic.go:356] "Generic (PLEG): container finished" podID="187edd17-e602-4707-8a36-7ee14162b6b9" containerID="7c32d1027191e4653023d2e9f521d7cfb956c25dbc3212dbf055d93ce91b9a8c" exitCode=0 Nov 02 21:54:29.300209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:29.299883 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" event={"ID":"187edd17-e602-4707-8a36-7ee14162b6b9","Type":"ContainerDied","Data":"7c32d1027191e4653023d2e9f521d7cfb956c25dbc3212dbf055d93ce91b9a8c"} Nov 02 21:54:30.478845 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.478815 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:30.563148 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.563078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js\" (UniqueName: \"kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js\") pod \"187edd17-e602-4707-8a36-7ee14162b6b9\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " Nov 02 21:54:30.563298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.563281 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca\") pod \"187edd17-e602-4707-8a36-7ee14162b6b9\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " Nov 02 21:54:30.563403 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.563387 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca\") pod \"187edd17-e602-4707-8a36-7ee14162b6b9\" (UID: \"187edd17-e602-4707-8a36-7ee14162b6b9\") " Nov 02 21:54:30.563702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.563662 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca" pod "187edd17-e602-4707-8a36-7ee14162b6b9" (UID: "187edd17-e602-4707-8a36-7ee14162b6b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:30.563825 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.563810 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:30.565209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.565185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js" pod "187edd17-e602-4707-8a36-7ee14162b6b9" (UID: "187edd17-e602-4707-8a36-7ee14162b6b9"). InnerVolumeSpecName "kube-api-access-f27js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:30.565316 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.565291 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca" pod "187edd17-e602-4707-8a36-7ee14162b6b9" (UID: "187edd17-e602-4707-8a36-7ee14162b6b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:30.665021 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.664987 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/187edd17-e602-4707-8a36-7ee14162b6b9-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:30.665021 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:30.665021 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f27js\" (UniqueName: \"kubernetes.io/projected/187edd17-e602-4707-8a36-7ee14162b6b9-kube-api-access-f27js\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:31.241134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.241053 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h"] Nov 02 21:54:31.241755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.241735 5091 cpu_manager.go:414] "containerMap: removing container" podUID="187edd17-e602-4707-8a36-7ee14162b6b9" containerName="revoke-keys" Nov 02 21:54:31.241755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.241755 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187edd17-e602-4707-8a36-7ee14162b6b9" containerName="revoke-keys" Nov 02 21:54:31.241908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.241893 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="187edd17-e602-4707-8a36-7ee14162b6b9" containerName="revoke-keys" Nov 02 21:54:31.245626 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.245602 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.320867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.320834 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" Nov 02 21:54:31.321025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.320873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk" event={"ID":"187edd17-e602-4707-8a36-7ee14162b6b9","Type":"ContainerDied","Data":"c0654ade2402f076bd158e3900a4f700069e4e00c157c3b69e446d37b50cf24b"} Nov 02 21:54:31.321025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.320933 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0654ade2402f076bd158e3900a4f700069e4e00c157c3b69e446d37b50cf24b" Nov 02 21:54:31.334339 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.334298 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h"] Nov 02 21:54:31.347129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.347104 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"distinct-2-w2-9cc56b-kserve-self-signed-certs\"" Nov 02 21:54:31.374721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.374659 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.374827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.374723 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.374827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.374812 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.374908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.374881 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.375011 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.374999 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.375055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.375042 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qw2\" (UniqueName: \"kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.480422 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.480378 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.481118 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.481091 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.481285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.481268 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qw2\" (UniqueName: \"kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.481485 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.481468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.481604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.481590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.481744 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.481729 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.484129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.483527 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.484129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.483835 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.484884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.484857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.487527 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.487505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.487801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.487776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.540435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.540354 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qw2\" (UniqueName: \"kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2\") pod \"distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.558310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.558279 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:31.946255 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:31.946227 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59f8c12f_cbce_4387_a9e6_f3a5e22796ee.slice/crio-e9414813b3b47d13d7f462406bb75eb1608f18f78c48ef772bbeac5dce24df59 WatchSource:0}: Error finding container e9414813b3b47d13d7f462406bb75eb1608f18f78c48ef772bbeac5dce24df59: Status 404 returned error can't find the container with id e9414813b3b47d13d7f462406bb75eb1608f18f78c48ef772bbeac5dce24df59 Nov 02 21:54:31.946872 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:31.946844 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h"] Nov 02 21:54:32.329242 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:32.329150 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" event={"ID":"59f8c12f-cbce-4387-a9e6-f3a5e22796ee","Type":"ContainerStarted","Data":"be54c3746766efacdcfdd160499fef54f2c3213f36e66114e22bbbe0fe1ea987"} Nov 02 21:54:32.329242 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:32.329203 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" event={"ID":"59f8c12f-cbce-4387-a9e6-f3a5e22796ee","Type":"ContainerStarted","Data":"e9414813b3b47d13d7f462406bb75eb1608f18f78c48ef772bbeac5dce24df59"} Nov 02 21:54:32.445551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:32.445496 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podStartSLOduration=1.4454844 podStartE2EDuration="1.4454844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:32.441634072 +0000 UTC m=+1219.603729192" watchObservedRunningTime="2026-11-02 21:54:32.4454844 +0000 UTC m=+1219.607579569" Nov 02 21:54:32.558927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:32.558876 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:35.051652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:35.051615 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4"] Nov 02 21:54:35.101302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:35.101266 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd1404c-m8pc4"] Nov 02 21:54:35.993088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:35.993052 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz"] Nov 02 21:54:35.997349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:35.997333 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.052534 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.052506 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz"] Nov 02 21:54:36.097428 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.097403 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:36.097428 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.097415 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:36.097621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.097481 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:36.098774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.098754 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:54:36.135333 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.135309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866tr\" (UniqueName: \"kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.135470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.135356 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.135704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.135681 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.236791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.236758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.236937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.236821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866tr\" (UniqueName: \"kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.236988 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.236976 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.237576 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.237551 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.239180 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.239150 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.293169 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.293116 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866tr\" (UniqueName: \"kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr\") pod \"maas-api-revoke-sub-e57de505-n7rlz\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.311069 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.311049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:36.683348 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:36.683314 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c353267_d739_4ffd_b939_57ce1baf09bf.slice/crio-4e4d6e90c6564001e91f4b6b2029cd1cfc9e25cad37431bec426bda19e7f5062 WatchSource:0}: Error finding container 4e4d6e90c6564001e91f4b6b2029cd1cfc9e25cad37431bec426bda19e7f5062: Status 404 returned error can't find the container with id 4e4d6e90c6564001e91f4b6b2029cd1cfc9e25cad37431bec426bda19e7f5062 Nov 02 21:54:36.683932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.683867 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz"] Nov 02 21:54:36.977561 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:36.977533 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff05d887-ddf1-4d71-a63c-326cabd4b1d6" path="/var/lib/kubelet/pods/ff05d887-ddf1-4d71-a63c-326cabd4b1d6/volumes" Nov 02 21:54:37.113349 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:54:37.113312 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:54:37.369111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:37.369078 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" event={"ID":"3c353267-d739-4ffd-b939-57ce1baf09bf","Type":"ContainerStarted","Data":"34d08c8054adcb62f115e3427b18abbe1610d31083f179560e41b56d0d36d35e"} Nov 02 21:54:37.369255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:37.369131 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" event={"ID":"3c353267-d739-4ffd-b939-57ce1baf09bf","Type":"ContainerStarted","Data":"4e4d6e90c6564001e91f4b6b2029cd1cfc9e25cad37431bec426bda19e7f5062"} Nov 02 21:54:37.483857 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:37.483807 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" podStartSLOduration=2.4837925419999998 podStartE2EDuration="2.483792542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:37.479908537 +0000 UTC m=+1224.642003660" watchObservedRunningTime="2026-11-02 21:54:37.483792542 +0000 UTC m=+1224.645887661" Nov 02 21:54:38.377484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:38.377450 5091 generic.go:356] "Generic (PLEG): container finished" podID="3c353267-d739-4ffd-b939-57ce1baf09bf" containerID="34d08c8054adcb62f115e3427b18abbe1610d31083f179560e41b56d0d36d35e" exitCode=0 Nov 02 21:54:38.377989 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:38.377534 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" event={"ID":"3c353267-d739-4ffd-b939-57ce1baf09bf","Type":"ContainerDied","Data":"34d08c8054adcb62f115e3427b18abbe1610d31083f179560e41b56d0d36d35e"} Nov 02 21:54:39.541003 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.540975 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:39.574354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.574325 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca\") pod \"3c353267-d739-4ffd-b939-57ce1baf09bf\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " Nov 02 21:54:39.574477 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.574461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr\" (UniqueName: \"kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr\") pod \"3c353267-d739-4ffd-b939-57ce1baf09bf\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " Nov 02 21:54:39.574517 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.574486 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca\") pod \"3c353267-d739-4ffd-b939-57ce1baf09bf\" (UID: \"3c353267-d739-4ffd-b939-57ce1baf09bf\") " Nov 02 21:54:39.574966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.574933 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca" pod "3c353267-d739-4ffd-b939-57ce1baf09bf" (UID: "3c353267-d739-4ffd-b939-57ce1baf09bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:39.576446 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.576419 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca" pod "3c353267-d739-4ffd-b939-57ce1baf09bf" (UID: "3c353267-d739-4ffd-b939-57ce1baf09bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:39.576699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.576642 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr" pod "3c353267-d739-4ffd-b939-57ce1baf09bf" (UID: "3c353267-d739-4ffd-b939-57ce1baf09bf"). InnerVolumeSpecName "kube-api-access-866tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:39.676377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.676302 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:39.676377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.676340 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c353267-d739-4ffd-b939-57ce1baf09bf-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:39.676377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:39.676357 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866tr\" (UniqueName: \"kubernetes.io/projected/3c353267-d739-4ffd-b939-57ce1baf09bf-kube-api-access-866tr\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:40.392664 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:40.392630 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" Nov 02 21:54:40.392875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:40.392665 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz" event={"ID":"3c353267-d739-4ffd-b939-57ce1baf09bf","Type":"ContainerDied","Data":"4e4d6e90c6564001e91f4b6b2029cd1cfc9e25cad37431bec426bda19e7f5062"} Nov 02 21:54:40.392875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:40.392734 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e4d6e90c6564001e91f4b6b2029cd1cfc9e25cad37431bec426bda19e7f5062" Nov 02 21:54:41.558735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:41.558704 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:41.571301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:41.571277 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:42.419302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:42.419265 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:54:43.286238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.286189 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b"] Nov 02 21:54:43.287785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.287756 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3c353267-d739-4ffd-b939-57ce1baf09bf" containerName="revoke-keys" Nov 02 21:54:43.287785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.287783 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c353267-d739-4ffd-b939-57ce1baf09bf" containerName="revoke-keys" Nov 02 21:54:43.288106 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.288090 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c353267-d739-4ffd-b939-57ce1baf09bf" containerName="revoke-keys" Nov 02 21:54:43.293579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.293528 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.327181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.327150 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpml\" (UniqueName: \"kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.327314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.327295 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.327426 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.327407 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.344130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.344098 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b"] Nov 02 21:54:43.388401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.388373 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:43.388592 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.388464 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:43.388706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.388518 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:54:43.396102 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.396080 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:43.429145 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.429118 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.429266 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.429179 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpml\" (UniqueName: \"kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.429337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.429295 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.429796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.429771 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.431919 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.431897 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.487886 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.487850 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpml\" (UniqueName: \"kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml\") pod \"maas-api-revoke-sub-11bff44a-wcv6b\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.612033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.611945 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:43.994272 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:43.994237 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3763db08_91c7_4b93_b380_30c9e5bd0248.slice/crio-741d0ca809cd469d43c529ac69248e8e9e4636dd3e0d650ab1649ed56219ba92 WatchSource:0}: Error finding container 741d0ca809cd469d43c529ac69248e8e9e4636dd3e0d650ab1649ed56219ba92: Status 404 returned error can't find the container with id 741d0ca809cd469d43c529ac69248e8e9e4636dd3e0d650ab1649ed56219ba92 Nov 02 21:54:43.995073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:43.995048 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b"] Nov 02 21:54:44.430126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:44.430091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" event={"ID":"3763db08-91c7-4b93-b380-30c9e5bd0248","Type":"ContainerStarted","Data":"7cafa5af39a260d591ff47f4bad6f98414d7c90a50b68edd5cd0b03f34080394"} Nov 02 21:54:44.430555 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:44.430146 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" event={"ID":"3763db08-91c7-4b93-b380-30c9e5bd0248","Type":"ContainerStarted","Data":"741d0ca809cd469d43c529ac69248e8e9e4636dd3e0d650ab1649ed56219ba92"} Nov 02 21:54:44.544626 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:44.544523 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" podStartSLOduration=1.544508602 podStartE2EDuration="1.544508602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:44.540987675 +0000 UTC m=+1231.703082795" watchObservedRunningTime="2026-11-02 21:54:44.544508602 +0000 UTC m=+1231.706603722" Nov 02 21:54:45.439647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:45.439615 5091 generic.go:356] "Generic (PLEG): container finished" podID="3763db08-91c7-4b93-b380-30c9e5bd0248" containerID="7cafa5af39a260d591ff47f4bad6f98414d7c90a50b68edd5cd0b03f34080394" exitCode=0 Nov 02 21:54:45.440038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:45.439710 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" event={"ID":"3763db08-91c7-4b93-b380-30c9e5bd0248","Type":"ContainerDied","Data":"7cafa5af39a260d591ff47f4bad6f98414d7c90a50b68edd5cd0b03f34080394"} Nov 02 21:54:46.598388 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.598359 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:46.667739 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.667709 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca\") pod \"3763db08-91c7-4b93-b380-30c9e5bd0248\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " Nov 02 21:54:46.667874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.667752 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca\") pod \"3763db08-91c7-4b93-b380-30c9e5bd0248\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " Nov 02 21:54:46.667874 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.667784 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml\" (UniqueName: \"kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml\") pod \"3763db08-91c7-4b93-b380-30c9e5bd0248\" (UID: \"3763db08-91c7-4b93-b380-30c9e5bd0248\") " Nov 02 21:54:46.668094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.668070 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca" pod "3763db08-91c7-4b93-b380-30c9e5bd0248" (UID: "3763db08-91c7-4b93-b380-30c9e5bd0248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:46.668258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.668238 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:46.669863 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.669841 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca" pod "3763db08-91c7-4b93-b380-30c9e5bd0248" (UID: "3763db08-91c7-4b93-b380-30c9e5bd0248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:46.669955 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.669908 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml" pod "3763db08-91c7-4b93-b380-30c9e5bd0248" (UID: "3763db08-91c7-4b93-b380-30c9e5bd0248"). InnerVolumeSpecName "kube-api-access-7xpml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:46.764233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.764149 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j"] Nov 02 21:54:46.764810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.764790 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3763db08-91c7-4b93-b380-30c9e5bd0248" containerName="revoke-keys" Nov 02 21:54:46.764939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.764810 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3763db08-91c7-4b93-b380-30c9e5bd0248" containerName="revoke-keys" Nov 02 21:54:46.765037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.765022 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3763db08-91c7-4b93-b380-30c9e5bd0248" containerName="revoke-keys" Nov 02 21:54:46.768721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.768699 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.769535 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.769512 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763db08-91c7-4b93-b380-30c9e5bd0248-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:46.769617 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.769541 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpml\" (UniqueName: \"kubernetes.io/projected/3763db08-91c7-4b93-b380-30c9e5bd0248-kube-api-access-7xpml\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:46.826044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.826013 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j"] Nov 02 21:54:46.870696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.870636 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtmw\" (UniqueName: \"kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.870851 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.870769 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.870851 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.870843 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.974066 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.974028 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.974432 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.974411 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtmw\" (UniqueName: \"kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.974610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.974593 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.982456 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.982427 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:46.982576 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:46.982466 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:47.038261 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:47.038207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtmw\" (UniqueName: \"kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw\") pod \"maas-api-revoke-sub-1819b8d4-ccp4j\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:47.082644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:47.082619 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:47.458274 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:47.458239 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacee79a2_7515_4de1_b7fa_f181ac2e9a14.slice/crio-a1f8d449217d96196803e3961df769779890279c9d1397ca19a65d0f93f1bea7 WatchSource:0}: Error finding container a1f8d449217d96196803e3961df769779890279c9d1397ca19a65d0f93f1bea7: Status 404 returned error can't find the container with id a1f8d449217d96196803e3961df769779890279c9d1397ca19a65d0f93f1bea7 Nov 02 21:54:47.458733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:47.458707 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" event={"ID":"3763db08-91c7-4b93-b380-30c9e5bd0248","Type":"ContainerDied","Data":"741d0ca809cd469d43c529ac69248e8e9e4636dd3e0d650ab1649ed56219ba92"} Nov 02 21:54:47.458860 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:47.458751 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b" Nov 02 21:54:47.458860 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:47.458756 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="741d0ca809cd469d43c529ac69248e8e9e4636dd3e0d650ab1649ed56219ba92" Nov 02 21:54:47.460013 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:47.459989 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j"] Nov 02 21:54:48.483169 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:48.483129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" event={"ID":"acee79a2-7515-4de1-b7fa-f181ac2e9a14","Type":"ContainerStarted","Data":"6413083ff1cdd564d58decf10ec479133ee38901385d1872f0592376c6e44217"} Nov 02 21:54:48.483888 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:48.483809 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" event={"ID":"acee79a2-7515-4de1-b7fa-f181ac2e9a14","Type":"ContainerStarted","Data":"a1f8d449217d96196803e3961df769779890279c9d1397ca19a65d0f93f1bea7"} Nov 02 21:54:48.602734 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:48.602575 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" podStartSLOduration=2.6025536320000002 podStartE2EDuration="2.602553632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:48.599059503 +0000 UTC m=+1235.761154623" watchObservedRunningTime="2026-11-02 21:54:48.602553632 +0000 UTC m=+1235.764648753" Nov 02 21:54:49.511691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:49.510895 5091 generic.go:356] "Generic (PLEG): container finished" podID="acee79a2-7515-4de1-b7fa-f181ac2e9a14" containerID="6413083ff1cdd564d58decf10ec479133ee38901385d1872f0592376c6e44217" exitCode=0 Nov 02 21:54:49.511691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:49.511032 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" event={"ID":"acee79a2-7515-4de1-b7fa-f181ac2e9a14","Type":"ContainerDied","Data":"6413083ff1cdd564d58decf10ec479133ee38901385d1872f0592376c6e44217"} Nov 02 21:54:50.070695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.069758 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.189872 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.189775 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.204837 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.204792 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:54:50.205119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.205073 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.282900 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.282855 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.436803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.436751 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:54:50.984330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:50.984303 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:51.078074 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.077965 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca\") pod \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " Nov 02 21:54:51.078235 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.078201 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw\" (UniqueName: \"kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw\") pod \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " Nov 02 21:54:51.078330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.078313 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca\") pod \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\" (UID: \"acee79a2-7515-4de1-b7fa-f181ac2e9a14\") " Nov 02 21:54:51.079932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.079901 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca" pod "acee79a2-7515-4de1-b7fa-f181ac2e9a14" (UID: "acee79a2-7515-4de1-b7fa-f181ac2e9a14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:51.082622 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.082583 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca" pod "acee79a2-7515-4de1-b7fa-f181ac2e9a14" (UID: "acee79a2-7515-4de1-b7fa-f181ac2e9a14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:51.090658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.090135 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw" pod "acee79a2-7515-4de1-b7fa-f181ac2e9a14" (UID: "acee79a2-7515-4de1-b7fa-f181ac2e9a14"). InnerVolumeSpecName "kube-api-access-cwtmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:51.186919 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.180851 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtmw\" (UniqueName: \"kubernetes.io/projected/acee79a2-7515-4de1-b7fa-f181ac2e9a14-kube-api-access-cwtmw\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:51.186919 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.180894 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:51.186919 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.180910 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acee79a2-7515-4de1-b7fa-f181ac2e9a14-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:54:51.542126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.542098 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" Nov 02 21:54:51.542313 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.542137 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j" event={"ID":"acee79a2-7515-4de1-b7fa-f181ac2e9a14","Type":"ContainerDied","Data":"a1f8d449217d96196803e3961df769779890279c9d1397ca19a65d0f93f1bea7"} Nov 02 21:54:51.542313 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:51.542185 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f8d449217d96196803e3961df769779890279c9d1397ca19a65d0f93f1bea7" Nov 02 21:54:53.050703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:53.050653 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq"] Nov 02 21:54:53.099950 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:53.099919 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b806723-s8ngq"] Nov 02 21:54:54.977420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:54.977388 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a2aea8-632e-4092-ad4f-b0949d66a3ec" path="/var/lib/kubelet/pods/d4a2aea8-632e-4092-ad4f-b0949d66a3ec/volumes" Nov 02 21:54:58.928342 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.928309 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2"] Nov 02 21:54:58.928901 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.928886 5091 cpu_manager.go:414] "containerMap: removing container" podUID="acee79a2-7515-4de1-b7fa-f181ac2e9a14" containerName="revoke-keys" Nov 02 21:54:58.928960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.928902 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acee79a2-7515-4de1-b7fa-f181ac2e9a14" containerName="revoke-keys" Nov 02 21:54:58.929027 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.929016 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="acee79a2-7515-4de1-b7fa-f181ac2e9a14" containerName="revoke-keys" Nov 02 21:54:58.933016 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.932997 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:58.970504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.970473 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:58.970739 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.970517 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:58.970739 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.970694 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9769f\" (UniqueName: \"kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:58.987778 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:58.987752 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2"] Nov 02 21:54:59.032126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.032097 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:54:59.032126 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.032106 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:54:59.034114 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.034095 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:54:59.034114 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.034109 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:54:59.071498 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.071463 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.071628 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.071502 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.071628 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.071575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9769f\" (UniqueName: \"kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.072292 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.072272 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.073959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.073942 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.127762 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.127730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9769f\" (UniqueName: \"kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f\") pod \"maas-api-revoke-sub-72d134b0-qrfx2\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.247166 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.247107 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:54:59.620291 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:54:59.620256 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff220e65_774a_44e9_85e8_85514e205eaa.slice/crio-82a8e015dfaf58c73a19d169bcb1e52b4adf28cc647b7dd06d5232a544d4a13d WatchSource:0}: Error finding container 82a8e015dfaf58c73a19d169bcb1e52b4adf28cc647b7dd06d5232a544d4a13d: Status 404 returned error can't find the container with id 82a8e015dfaf58c73a19d169bcb1e52b4adf28cc647b7dd06d5232a544d4a13d Nov 02 21:54:59.622028 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:54:59.622003 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2"] Nov 02 21:55:00.624560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:00.624498 5091 generic.go:356] "Generic (PLEG): container finished" podID="ff220e65-774a-44e9-85e8-85514e205eaa" containerID="d123db565d1c258aa29a19515692a2d6c2b1742a3487a909c875cb13655edfbb" exitCode=0 Nov 02 21:55:00.624957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:00.624584 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" event={"ID":"ff220e65-774a-44e9-85e8-85514e205eaa","Type":"ContainerDied","Data":"d123db565d1c258aa29a19515692a2d6c2b1742a3487a909c875cb13655edfbb"} Nov 02 21:55:00.624957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:00.624662 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" event={"ID":"ff220e65-774a-44e9-85e8-85514e205eaa","Type":"ContainerStarted","Data":"82a8e015dfaf58c73a19d169bcb1e52b4adf28cc647b7dd06d5232a544d4a13d"} Nov 02 21:55:01.789867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.789842 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:55:01.904740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.904630 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca\") pod \"ff220e65-774a-44e9-85e8-85514e205eaa\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " Nov 02 21:55:01.904740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.904717 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f\" (UniqueName: \"kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f\") pod \"ff220e65-774a-44e9-85e8-85514e205eaa\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " Nov 02 21:55:01.904977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.904959 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca\") pod \"ff220e65-774a-44e9-85e8-85514e205eaa\" (UID: \"ff220e65-774a-44e9-85e8-85514e205eaa\") " Nov 02 21:55:01.905573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.905537 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca" pod "ff220e65-774a-44e9-85e8-85514e205eaa" (UID: "ff220e65-774a-44e9-85e8-85514e205eaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:01.907204 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.907169 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca" pod "ff220e65-774a-44e9-85e8-85514e205eaa" (UID: "ff220e65-774a-44e9-85e8-85514e205eaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:01.907204 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:01.907192 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f" pod "ff220e65-774a-44e9-85e8-85514e205eaa" (UID: "ff220e65-774a-44e9-85e8-85514e205eaa"). InnerVolumeSpecName "kube-api-access-9769f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:02.007073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.007039 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:02.007073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.007071 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff220e65-774a-44e9-85e8-85514e205eaa-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:02.007248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.007085 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9769f\" (UniqueName: \"kubernetes.io/projected/ff220e65-774a-44e9-85e8-85514e205eaa-kube-api-access-9769f\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:02.105411 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.105376 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f"] Nov 02 21:55:02.106070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.106049 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ff220e65-774a-44e9-85e8-85514e205eaa" containerName="revoke-keys" Nov 02 21:55:02.106070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.106070 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff220e65-774a-44e9-85e8-85514e205eaa" containerName="revoke-keys" Nov 02 21:55:02.106250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.106217 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff220e65-774a-44e9-85e8-85514e205eaa" containerName="revoke-keys" Nov 02 21:55:02.110122 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.110097 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.165081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.165020 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f"] Nov 02 21:55:02.208656 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.208628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrbt\" (UniqueName: \"kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.208827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.208683 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.208827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.208788 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.310505 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.310476 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrbt\" (UniqueName: \"kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.310663 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.310518 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.310663 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.310556 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.311228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.311202 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.312855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.312835 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.367618 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.367587 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrbt\" (UniqueName: \"kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt\") pod \"maas-api-revoke-sub-f09e756c-8fb5f\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.424864 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.424800 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:02.641348 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.641318 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" Nov 02 21:55:02.641514 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.641319 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2" event={"ID":"ff220e65-774a-44e9-85e8-85514e205eaa","Type":"ContainerDied","Data":"82a8e015dfaf58c73a19d169bcb1e52b4adf28cc647b7dd06d5232a544d4a13d"} Nov 02 21:55:02.641514 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.641446 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a8e015dfaf58c73a19d169bcb1e52b4adf28cc647b7dd06d5232a544d4a13d" Nov 02 21:55:02.803466 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:02.803438 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5b7e6c_af5c_485c_9fe6_1fbebba89a9d.slice/crio-70876c7090d624f3a4d76bc63ab7d7678c5450eb2a9037693827ec06973c63ee WatchSource:0}: Error finding container 70876c7090d624f3a4d76bc63ab7d7678c5450eb2a9037693827ec06973c63ee: Status 404 returned error can't find the container with id 70876c7090d624f3a4d76bc63ab7d7678c5450eb2a9037693827ec06973c63ee Nov 02 21:55:02.805225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:02.805197 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f"] Nov 02 21:55:03.650128 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:03.650092 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" event={"ID":"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d","Type":"ContainerStarted","Data":"13216b39182b17efd066f68663248d5033977993b42cd5ce0c1e923b6cb76c8e"} Nov 02 21:55:03.650300 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:03.650152 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" event={"ID":"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d","Type":"ContainerStarted","Data":"70876c7090d624f3a4d76bc63ab7d7678c5450eb2a9037693827ec06973c63ee"} Nov 02 21:55:03.765047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:03.765003 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" podStartSLOduration=1.764989403 podStartE2EDuration="1.764989403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:03.760849302 +0000 UTC m=+1250.922944423" watchObservedRunningTime="2026-11-02 21:55:03.764989403 +0000 UTC m=+1250.927084522" Nov 02 21:55:04.658559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:04.658524 5091 generic.go:356] "Generic (PLEG): container finished" podID="5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" containerID="13216b39182b17efd066f68663248d5033977993b42cd5ce0c1e923b6cb76c8e" exitCode=0 Nov 02 21:55:04.658994 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:04.658591 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" event={"ID":"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d","Type":"ContainerDied","Data":"13216b39182b17efd066f68663248d5033977993b42cd5ce0c1e923b6cb76c8e"} Nov 02 21:55:05.834192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.834167 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:05.948987 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.948954 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca\") pod \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " Nov 02 21:55:05.949159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.949030 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt\" (UniqueName: \"kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt\") pod \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " Nov 02 21:55:05.949159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.949091 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca\") pod \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\" (UID: \"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d\") " Nov 02 21:55:05.949340 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.949313 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca" pod "5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" (UID: "5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:05.949682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.949645 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:05.951513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.951487 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt" pod "5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" (UID: "5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d"). InnerVolumeSpecName "kube-api-access-5zrbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:05.951720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:05.951691 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca" pod "5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" (UID: "5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:06.027866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.027795 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz"] Nov 02 21:55:06.028401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.028384 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" containerName="revoke-keys" Nov 02 21:55:06.028401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.028402 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" containerName="revoke-keys" Nov 02 21:55:06.028535 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.028526 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" containerName="revoke-keys" Nov 02 21:55:06.031784 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.031769 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.047993 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.047969 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5"] Nov 02 21:55:06.050960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.050931 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrbt\" (UniqueName: \"kubernetes.io/projected/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-kube-api-access-5zrbt\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.051068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.050962 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.052267 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.052247 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd65be97-nmsr5"] Nov 02 21:55:06.087496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.087470 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz"] Nov 02 21:55:06.152441 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.152370 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.152596 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.152538 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.152596 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.152570 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjb5\" (UniqueName: \"kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.253552 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.253522 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.253733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.253697 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.253820 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.253746 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjb5\" (UniqueName: \"kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.254190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.254163 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.255925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.255899 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.309519 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.309462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjb5\" (UniqueName: \"kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5\") pod \"maas-api-revoke-sub-c49c481d-4rmjz\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.346006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.345974 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:06.675686 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.675656 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" Nov 02 21:55:06.675858 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.675699 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f" event={"ID":"5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d","Type":"ContainerDied","Data":"70876c7090d624f3a4d76bc63ab7d7678c5450eb2a9037693827ec06973c63ee"} Nov 02 21:55:06.675858 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.675745 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70876c7090d624f3a4d76bc63ab7d7678c5450eb2a9037693827ec06973c63ee" Nov 02 21:55:06.719646 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:06.719620 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3af708_2288_4621_a7ed_535e1700ca97.slice/crio-2d6323c000086012c442fd938cb80fb97f3a6d2e29215029546a15f09df5d7f3 WatchSource:0}: Error finding container 2d6323c000086012c442fd938cb80fb97f3a6d2e29215029546a15f09df5d7f3: Status 404 returned error can't find the container with id 2d6323c000086012c442fd938cb80fb97f3a6d2e29215029546a15f09df5d7f3 Nov 02 21:55:06.720470 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.720448 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz"] Nov 02 21:55:06.978022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:06.977991 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239deac6-670a-4b0a-a61b-f34af8bc8b15" path="/var/lib/kubelet/pods/239deac6-670a-4b0a-a61b-f34af8bc8b15/volumes" Nov 02 21:55:07.112205 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:55:07.112168 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:55:07.236866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.236793 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz"] Nov 02 21:55:07.241327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.241307 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.297857 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.297823 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz"] Nov 02 21:55:07.367630 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.367602 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nczrd\" (UniqueName: \"kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.367807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.367660 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.367862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.367837 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.469037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.469004 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nczrd\" (UniqueName: \"kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.469202 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.469072 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.469202 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.469193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.469847 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.469821 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.471413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.471389 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.524953 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.524884 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nczrd\" (UniqueName: \"kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd\") pod \"maas-api-revoke-sub-0822b887-c7zsz\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.554756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.554732 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:07.691880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.691846 5091 generic.go:356] "Generic (PLEG): container finished" podID="6e3af708-2288-4621-a7ed-535e1700ca97" containerID="0f0e8266c96d717d41783b2235746c0d9707b5e8e92e0fc06064e9f225bc77bd" exitCode=0 Nov 02 21:55:07.692040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.691930 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" event={"ID":"6e3af708-2288-4621-a7ed-535e1700ca97","Type":"ContainerDied","Data":"0f0e8266c96d717d41783b2235746c0d9707b5e8e92e0fc06064e9f225bc77bd"} Nov 02 21:55:07.692040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.691980 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" event={"ID":"6e3af708-2288-4621-a7ed-535e1700ca97","Type":"ContainerStarted","Data":"2d6323c000086012c442fd938cb80fb97f3a6d2e29215029546a15f09df5d7f3"} Nov 02 21:55:07.927923 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:07.927883 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730350df_3b31_44d5_8592_604f25e4917a.slice/crio-3c88c1ec15c6f3430780f31e39b578241de64f4852f8c74500f1f4e89102e9c4 WatchSource:0}: Error finding container 3c88c1ec15c6f3430780f31e39b578241de64f4852f8c74500f1f4e89102e9c4: Status 404 returned error can't find the container with id 3c88c1ec15c6f3430780f31e39b578241de64f4852f8c74500f1f4e89102e9c4 Nov 02 21:55:07.928601 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:07.928580 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz"] Nov 02 21:55:08.045619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.045591 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2"] Nov 02 21:55:08.049760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.049735 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59308df-bpdd2"] Nov 02 21:55:08.701934 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.701897 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" event={"ID":"730350df-3b31-44d5-8592-604f25e4917a","Type":"ContainerStarted","Data":"1b5277c04c6b4146456058f3f7f03c5ada1e560d4667235060ef19c10b281aa0"} Nov 02 21:55:08.702098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.701954 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" event={"ID":"730350df-3b31-44d5-8592-604f25e4917a","Type":"ContainerStarted","Data":"3c88c1ec15c6f3430780f31e39b578241de64f4852f8c74500f1f4e89102e9c4"} Nov 02 21:55:08.817510 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.817444 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" podStartSLOduration=1.817415686 podStartE2EDuration="1.817415686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:08.814086537 +0000 UTC m=+1255.976181661" watchObservedRunningTime="2026-11-02 21:55:08.817415686 +0000 UTC m=+1255.979510821" Nov 02 21:55:08.857858 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.857833 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:08.980937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.980868 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a448efd3-f9a1-435d-b777-70922943cf73" path="/var/lib/kubelet/pods/a448efd3-f9a1-435d-b777-70922943cf73/volumes" Nov 02 21:55:08.990772 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.990739 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca\") pod \"6e3af708-2288-4621-a7ed-535e1700ca97\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " Nov 02 21:55:08.990939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.990910 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca\") pod \"6e3af708-2288-4621-a7ed-535e1700ca97\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " Nov 02 21:55:08.991011 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.990946 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5\" (UniqueName: \"kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5\") pod \"6e3af708-2288-4621-a7ed-535e1700ca97\" (UID: \"6e3af708-2288-4621-a7ed-535e1700ca97\") " Nov 02 21:55:08.992006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.991976 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca" pod "6e3af708-2288-4621-a7ed-535e1700ca97" (UID: "6e3af708-2288-4621-a7ed-535e1700ca97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:08.994902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:08.994863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5" pod "6e3af708-2288-4621-a7ed-535e1700ca97" (UID: "6e3af708-2288-4621-a7ed-535e1700ca97"). InnerVolumeSpecName "kube-api-access-rcjb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:09.000068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.000043 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca" pod "6e3af708-2288-4621-a7ed-535e1700ca97" (UID: "6e3af708-2288-4621-a7ed-535e1700ca97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:09.092189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.092162 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjb5\" (UniqueName: \"kubernetes.io/projected/6e3af708-2288-4621-a7ed-535e1700ca97-kube-api-access-rcjb5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:09.092189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.092186 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:09.092641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.092205 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3af708-2288-4621-a7ed-535e1700ca97-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:09.711394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.711363 5091 generic.go:356] "Generic (PLEG): container finished" podID="730350df-3b31-44d5-8592-604f25e4917a" containerID="1b5277c04c6b4146456058f3f7f03c5ada1e560d4667235060ef19c10b281aa0" exitCode=0 Nov 02 21:55:09.711575 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.711452 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" event={"ID":"730350df-3b31-44d5-8592-604f25e4917a","Type":"ContainerDied","Data":"1b5277c04c6b4146456058f3f7f03c5ada1e560d4667235060ef19c10b281aa0"} Nov 02 21:55:09.712919 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.712894 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" Nov 02 21:55:09.713030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.712942 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz" event={"ID":"6e3af708-2288-4621-a7ed-535e1700ca97","Type":"ContainerDied","Data":"2d6323c000086012c442fd938cb80fb97f3a6d2e29215029546a15f09df5d7f3"} Nov 02 21:55:09.713030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:09.712988 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6323c000086012c442fd938cb80fb97f3a6d2e29215029546a15f09df5d7f3" Nov 02 21:55:10.873633 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:10.873609 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:11.012351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.012270 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca\") pod \"730350df-3b31-44d5-8592-604f25e4917a\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " Nov 02 21:55:11.012514 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.012366 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd\" (UniqueName: \"kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd\") pod \"730350df-3b31-44d5-8592-604f25e4917a\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " Nov 02 21:55:11.012514 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.012436 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca\") pod \"730350df-3b31-44d5-8592-604f25e4917a\" (UID: \"730350df-3b31-44d5-8592-604f25e4917a\") " Nov 02 21:55:11.012806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.012780 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca" pod "730350df-3b31-44d5-8592-604f25e4917a" (UID: "730350df-3b31-44d5-8592-604f25e4917a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:11.014494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.014467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd" pod "730350df-3b31-44d5-8592-604f25e4917a" (UID: "730350df-3b31-44d5-8592-604f25e4917a"). InnerVolumeSpecName "kube-api-access-nczrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:11.014892 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.014864 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca" pod "730350df-3b31-44d5-8592-604f25e4917a" (UID: "730350df-3b31-44d5-8592-604f25e4917a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:11.114268 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.114229 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730350df-3b31-44d5-8592-604f25e4917a-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:11.114501 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.114480 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nczrd\" (UniqueName: \"kubernetes.io/projected/730350df-3b31-44d5-8592-604f25e4917a-kube-api-access-nczrd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:11.114572 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.114507 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730350df-3b31-44d5-8592-604f25e4917a-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:11.728696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.728649 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" Nov 02 21:55:11.728878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.728645 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz" event={"ID":"730350df-3b31-44d5-8592-604f25e4917a","Type":"ContainerDied","Data":"3c88c1ec15c6f3430780f31e39b578241de64f4852f8c74500f1f4e89102e9c4"} Nov 02 21:55:11.728878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.728780 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c88c1ec15c6f3430780f31e39b578241de64f4852f8c74500f1f4e89102e9c4" Nov 02 21:55:11.789900 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.789869 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn"] Nov 02 21:55:11.790459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.790444 5091 cpu_manager.go:414] "containerMap: removing container" podUID="730350df-3b31-44d5-8592-604f25e4917a" containerName="revoke-keys" Nov 02 21:55:11.790507 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.790461 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730350df-3b31-44d5-8592-604f25e4917a" containerName="revoke-keys" Nov 02 21:55:11.790507 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.790499 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6e3af708-2288-4621-a7ed-535e1700ca97" containerName="revoke-keys" Nov 02 21:55:11.790507 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.790504 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3af708-2288-4621-a7ed-535e1700ca97" containerName="revoke-keys" Nov 02 21:55:11.790608 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.790601 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3af708-2288-4621-a7ed-535e1700ca97" containerName="revoke-keys" Nov 02 21:55:11.790642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.790614 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="730350df-3b31-44d5-8592-604f25e4917a" containerName="revoke-keys" Nov 02 21:55:11.794172 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.794158 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.922700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.922633 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qp8m\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.922755 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.922912 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.923083 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.923109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.923150 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.923183 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.923204 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.923381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.923246 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:11.939801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:11.939772 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn"] Nov 02 21:55:12.007558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.007479 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5"] Nov 02 21:55:12.011751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.011730 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.024313 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024288 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024422 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024558 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024615 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024642 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024702 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024854 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024721 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024854 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024790 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qp8m\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.024854 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.025019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.024889 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.025081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.025017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.025081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.025040 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.025209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.025188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.026803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.026787 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.027004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.026984 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.065471 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.065422 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-64438f-openshift-default-dockercfg-tx22g\"" Nov 02 21:55:12.072474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.072451 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5"] Nov 02 21:55:12.081104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.081078 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.082350 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.082323 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qp8m\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.107518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.107494 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:12.125641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125615 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.125811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125679 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.125811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125711 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.125811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125757 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.125811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.126063 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.126063 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.125907 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.126063 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.126046 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5ng\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.127043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.126785 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.227957 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.227922 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.227975 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228032 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228097 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228378 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228145 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228378 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228228 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5ng\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228505 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228483 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.228867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.228849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.229135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.229115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.229228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.229191 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.229228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.229220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.231357 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.231317 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.231466 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.231447 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.289001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.288939 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.289110 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.289018 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5ng\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.325453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.325423 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:12.535814 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:12.535784 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435841da_c8b8_4060_85a5_346e9746144e.slice/crio-8acc69310d91b204aca89a5703b5f877c160a4b54f1699fbb08ce08d6caf8583 WatchSource:0}: Error finding container 8acc69310d91b204aca89a5703b5f877c160a4b54f1699fbb08ce08d6caf8583: Status 404 returned error can't find the container with id 8acc69310d91b204aca89a5703b5f877c160a4b54f1699fbb08ce08d6caf8583 Nov 02 21:55:12.536111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.536089 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn"] Nov 02 21:55:12.538073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.538041 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.538169 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.538115 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.538169 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.538149 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.711630 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.711600 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5"] Nov 02 21:55:12.712244 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:12.712221 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df05dfa_a092_4271_a600_96dc2ce787dc.slice/crio-7559cc37f40bffd035c24ee301fe70ab783dad3e8981fb2ba9dcea1c3817981c WatchSource:0}: Error finding container 7559cc37f40bffd035c24ee301fe70ab783dad3e8981fb2ba9dcea1c3817981c: Status 404 returned error can't find the container with id 7559cc37f40bffd035c24ee301fe70ab783dad3e8981fb2ba9dcea1c3817981c Nov 02 21:55:12.714906 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.714874 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.714972 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.714941 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.715014 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.714972 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.738717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.738656 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" event={"ID":"6df05dfa-a092-4271-a600-96dc2ce787dc","Type":"ContainerStarted","Data":"7559cc37f40bffd035c24ee301fe70ab783dad3e8981fb2ba9dcea1c3817981c"} Nov 02 21:55:12.740578 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.740548 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" event={"ID":"435841da-c8b8-4060-85a5-346e9746144e","Type":"ContainerStarted","Data":"88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f"} Nov 02 21:55:12.740664 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.740597 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" event={"ID":"435841da-c8b8-4060-85a5-346e9746144e","Type":"ContainerStarted","Data":"8acc69310d91b204aca89a5703b5f877c160a4b54f1699fbb08ce08d6caf8583"} Nov 02 21:55:12.857020 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:12.856897 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podStartSLOduration=1.8568835240000001 podStartE2EDuration="1.856883524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:12.855477733 +0000 UTC m=+1260.017572866" watchObservedRunningTime="2026-11-02 21:55:12.856883524 +0000 UTC m=+1260.018978718" Nov 02 21:55:13.048759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.048601 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth"] Nov 02 21:55:13.053307 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.053282 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.116019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.115937 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth"] Nov 02 21:55:13.117363 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.116843 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:13.118388 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.118359 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:13.118496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.118406 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:13.140603 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140570 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.140762 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140609 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.140762 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140745 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.140897 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140783 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.140897 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140815 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.140897 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.141053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140909 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb26t\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.141053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140936 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.141053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.140952 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242333 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242383 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242487 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242594 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb26t\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242699 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242748 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242851 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242867 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.242944 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.242912 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.243319 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.243290 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.243439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.243307 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.243550 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.243308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.243845 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.243825 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.244817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.244796 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.245593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.245574 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.299151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.299112 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb26t\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.300327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.300304 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.376694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.374103 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:13.751420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.751341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" event={"ID":"6df05dfa-a092-4271-a600-96dc2ce787dc","Type":"ContainerStarted","Data":"b82b97be47f234259ab4d7a154454934f7506cd32bb7275ceb75115a4bcc5c03"} Nov 02 21:55:13.820815 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.820725 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth"] Nov 02 21:55:13.822779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.822736 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.822887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.822833 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.822927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.822902 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.916696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:13.916193 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podStartSLOduration=2.91615792 podStartE2EDuration="2.91615792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:13.886393422 +0000 UTC m=+1261.048488571" watchObservedRunningTime="2026-11-02 21:55:13.91615792 +0000 UTC m=+1261.078253043" Nov 02 21:55:14.121006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.120514 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:14.121006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.120583 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:14.225503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.225459 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j"] Nov 02 21:55:14.229847 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.229820 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.282696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.282443 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:55:14.282696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.282494 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:55:14.282696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.282565 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:55:14.282696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.282627 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:55:14.328731 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.326014 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:14.328927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.328726 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:14.328927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.328781 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:14.332820 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.330479 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j"] Nov 02 21:55:14.362516 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.362414 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.362516 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.362443 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sz4\" (UniqueName: \"kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.362763 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.362737 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.493755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.487499 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.493755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.487575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.493755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.487603 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sz4\" (UniqueName: \"kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.493755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.488462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.511694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.506150 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.544572 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.544544 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sz4\" (UniqueName: \"kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4\") pod \"maas-api-revoke-sub-9bc7ad71-54j7j\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.554583 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.554526 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:14.806575 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.806482 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" event={"ID":"c3506f66-8b05-4c38-aff7-2451d20b7296","Type":"ContainerStarted","Data":"ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba"} Nov 02 21:55:14.812322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.812295 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" event={"ID":"c3506f66-8b05-4c38-aff7-2451d20b7296","Type":"ContainerStarted","Data":"0b498a774e391892d4c2cf43d743a016795362fdd4932b1a25f273b7776303e6"} Nov 02 21:55:14.950065 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:14.949992 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podStartSLOduration=1.949970301 podStartE2EDuration="1.949970301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:14.942221861 +0000 UTC m=+1262.104316986" watchObservedRunningTime="2026-11-02 21:55:14.949970301 +0000 UTC m=+1262.112065424" Nov 02 21:55:15.053763 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.053736 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j"] Nov 02 21:55:15.056416 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:15.056389 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8cc440_57d5_4ca1_99c0_07a4b1f8262b.slice/crio-dbbcc0a9b2ef635e99f5822a82c214d93390d5558aef320c28c90ca0322e2e18 WatchSource:0}: Error finding container dbbcc0a9b2ef635e99f5822a82c214d93390d5558aef320c28c90ca0322e2e18: Status 404 returned error can't find the container with id dbbcc0a9b2ef635e99f5822a82c214d93390d5558aef320c28c90ca0322e2e18 Nov 02 21:55:15.145028 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.144997 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:15.145623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.145588 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:15.333754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.332102 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:15.333754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.332187 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:15.380714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.379550 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:15.380714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.379692 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:15.380714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.379738 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:15.845696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.844743 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" event={"ID":"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b","Type":"ContainerStarted","Data":"f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b"} Nov 02 21:55:15.845696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.844807 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" event={"ID":"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b","Type":"ContainerStarted","Data":"dbbcc0a9b2ef635e99f5822a82c214d93390d5558aef320c28c90ca0322e2e18"} Nov 02 21:55:15.958688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:15.958626 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" podStartSLOduration=1.958609061 podStartE2EDuration="1.958609061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:15.956801553 +0000 UTC m=+1263.118896675" watchObservedRunningTime="2026-11-02 21:55:15.958609061 +0000 UTC m=+1263.120704183" Nov 02 21:55:16.109752 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.109112 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:16.109752 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.109202 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:16.340271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.340230 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:16.340907 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.340866 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:16.380704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.377951 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:16.380704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.378015 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:16.644305 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:55:16.644190 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8cc440_57d5_4ca1_99c0_07a4b1f8262b.slice/crio-conmon-f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8cc440_57d5_4ca1_99c0_07a4b1f8262b.slice/crio-f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:55:16.644482 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:55:16.644314 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8cc440_57d5_4ca1_99c0_07a4b1f8262b.slice/crio-f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8cc440_57d5_4ca1_99c0_07a4b1f8262b.slice/crio-conmon-f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:55:16.854530 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.854500 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" containerID="f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b" exitCode=0 Nov 02 21:55:16.854832 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:16.854548 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" event={"ID":"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b","Type":"ContainerDied","Data":"f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b"} Nov 02 21:55:17.061563 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.061480 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl"] Nov 02 21:55:17.108148 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.108105 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:17.108398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.108365 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:17.151332 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.151298 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866b5736-xh5pl"] Nov 02 21:55:17.329292 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.329185 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:17.329590 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.329538 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:17.378213 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.378177 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:17.378855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:17.378796 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:18.127868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.125409 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:18.127868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.125488 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:18.326525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.326059 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:18.326525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.326127 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:18.379036 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.378936 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:18.379787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.379655 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:18.598693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.597050 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:18.649774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.647435 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca\") pod \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " Nov 02 21:55:18.649774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.647647 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4\" (UniqueName: \"kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4\") pod \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " Nov 02 21:55:18.658861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.657263 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca\") pod \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\" (UID: \"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b\") " Nov 02 21:55:18.658861 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.658136 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca" pod "8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" (UID: "8b8cc440-57d5-4ca1-99c0-07a4b1f8262b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:18.714546 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.708851 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4" pod "8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" (UID: "8b8cc440-57d5-4ca1-99c0-07a4b1f8262b"). InnerVolumeSpecName "kube-api-access-78sz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:18.714917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.714858 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca" pod "8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" (UID: "8b8cc440-57d5-4ca1-99c0-07a4b1f8262b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:18.859101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.817498 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78sz4\" (UniqueName: \"kubernetes.io/projected/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-kube-api-access-78sz4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:18.859101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.817539 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:18.859101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.817554 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:18.939554 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:18.939462 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:55:19.078696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.067107 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f946dfc7-847b-40c3-86ee-fa5fddc5bd35" path="/var/lib/kubelet/pods/f946dfc7-847b-40c3-86ee-fa5fddc5bd35/volumes" Nov 02 21:55:19.116751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.092978 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" event={"ID":"8b8cc440-57d5-4ca1-99c0-07a4b1f8262b","Type":"ContainerDied","Data":"dbbcc0a9b2ef635e99f5822a82c214d93390d5558aef320c28c90ca0322e2e18"} Nov 02 21:55:19.116751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.093043 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbcc0a9b2ef635e99f5822a82c214d93390d5558aef320c28c90ca0322e2e18" Nov 02 21:55:19.116751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.093111 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j" Nov 02 21:55:19.157702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.157110 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:19.157702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.157165 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:19.282560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.281802 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:19.327591 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.327555 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:19.327898 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.327870 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:19.386696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.385816 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:19.386696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:19.385910 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:20.118697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.116857 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:55:20.118697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.116915 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:55:20.329228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.329170 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:55:20.329569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.329512 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:55:20.375258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.375171 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:20.375528 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.375498 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:20.830001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.829860 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:20.934840 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:20.934792 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.102747 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.076789 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.148831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.147045 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.257695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.235310 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.281838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.281782 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.306695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.290774 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.474694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.474086 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:21.474694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.474164 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:21.477244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.477208 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:55:21.503696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.499472 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 21:55:21.748213 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.748118 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:21.748475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.748443 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.797229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.797185 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.839850 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.839804 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.840783 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.840758 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:55:21.980813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.980763 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.987381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:21.981397 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:55:22.213706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.213644 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.241060 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.215557 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:22.242508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.215914 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 21:55:22.250816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.243103 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:22.251033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.251017 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:55:22.265692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.253207 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.266210 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.266187 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:55:22.334519 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.334309 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.504785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.504416 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.531694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.506552 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.549140 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.537080 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.552577 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.553801 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.567935 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.568374 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.591759 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:22.592708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.591827 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.596898 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.556526 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.606221 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.606197 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:55:22.608697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.608638 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.849746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.848348 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:55:22.849746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.848411 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:55:22.867701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.854733 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:55:22.867701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.854791 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:55:22.867701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.854823 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:55:22.867701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.856740 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:55:22.957189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.953809 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:55:22.963631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.963161 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:22.967692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.964048 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:22.967692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:22.966850 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 21:55:23.432404 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.432336 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x"] Nov 02 21:55:23.433471 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.433441 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" containerName="revoke-keys" Nov 02 21:55:23.433471 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.433469 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" containerName="revoke-keys" Nov 02 21:55:23.433649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.433636 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" containerName="revoke-keys" Nov 02 21:55:23.440416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.440392 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.493002 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.492972 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:55:23.493924 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.493892 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x"] Nov 02 21:55:23.496000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.495845 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:55:23.496000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.495983 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:55:23.548825 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.548788 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.549000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.548835 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnt9b\" (UniqueName: \"kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.549000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.548933 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.592660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.592631 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:55:23.653427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.653387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.653745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.653459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnt9b\" (UniqueName: \"kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.653745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.653536 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.655552 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.655521 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.658298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.658268 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.710757 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.710636 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnt9b\" (UniqueName: \"kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b\") pod \"maas-api-revoke-sub-a011a95a-mkk6x\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:23.756466 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:23.756430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:24.149610 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:24.149578 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9c0a47_af7e_4075_a0d5_9ae6767e1d23.slice/crio-9900295fcae098bb00bdc4fa66f1f150b4bd4a29bc107537895e8d55e33edb88 WatchSource:0}: Error finding container 9900295fcae098bb00bdc4fa66f1f150b4bd4a29bc107537895e8d55e33edb88: Status 404 returned error can't find the container with id 9900295fcae098bb00bdc4fa66f1f150b4bd4a29bc107537895e8d55e33edb88 Nov 02 21:55:24.153227 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.153199 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x"] Nov 02 21:55:24.418888 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.418796 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:24.418888 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.418864 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:24.752445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.752365 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" event={"ID":"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23","Type":"ContainerStarted","Data":"a2f346eb3f3397195cefb0ff888959bd97bfc1017132f49e5c9a89ad8d717638"} Nov 02 21:55:24.752445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.752422 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" event={"ID":"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23","Type":"ContainerStarted","Data":"9900295fcae098bb00bdc4fa66f1f150b4bd4a29bc107537895e8d55e33edb88"} Nov 02 21:55:24.767296 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.767264 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:24.767760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.767729 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:24.768728 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.768707 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 21:55:24.868660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:24.868604 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" podStartSLOduration=1.868587562 podStartE2EDuration="1.868587562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:24.866956683 +0000 UTC m=+1272.029051804" watchObservedRunningTime="2026-11-02 21:55:24.868587562 +0000 UTC m=+1272.030682683" Nov 02 21:55:25.763130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:25.763096 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" containerID="a2f346eb3f3397195cefb0ff888959bd97bfc1017132f49e5c9a89ad8d717638" exitCode=0 Nov 02 21:55:25.763578 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:25.763180 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" event={"ID":"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23","Type":"ContainerDied","Data":"a2f346eb3f3397195cefb0ff888959bd97bfc1017132f49e5c9a89ad8d717638"} Nov 02 21:55:26.699439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.699405 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm"] Nov 02 21:55:26.702689 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.702654 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.760647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.760615 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm"] Nov 02 21:55:26.806584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.806551 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.807172 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.806753 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.807172 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.806788 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhjf\" (UniqueName: \"kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.862271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.862240 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c"] Nov 02 21:55:26.875698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.868774 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908155 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908199 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlb8\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908231 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908373 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908409 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.908490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908446 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908536 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908487 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908536 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908566 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.908699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908594 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhjf\" (UniqueName: \"kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.908699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.908699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.908691 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:26.909983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.909545 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.913661 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.913637 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.959975 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.959916 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c"] Nov 02 21:55:26.966959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.966937 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhjf\" (UniqueName: \"kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf\") pod \"maas-api-revoke-sub-1f060ba6-bz9tm\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:26.967890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:26.967870 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:27.010153 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010125 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca\") pod \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " Nov 02 21:55:27.010277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010201 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca\") pod \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " Nov 02 21:55:27.010351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010317 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b\" (UniqueName: \"kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b\") pod \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\" (UID: \"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23\") " Nov 02 21:55:27.010516 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010496 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.010605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.010658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010605 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.010658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca" pod "1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" (UID: "1c9c0a47-af7e-4075-a0d5-9ae6767e1d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:27.010810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010765 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.010810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010794 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlb8\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.010916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010826 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.010972 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010915 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011036 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010964 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011036 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.010977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011036 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.011022 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011276 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.011043 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011276 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.011154 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:27.011413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.011340 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011488 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.011380 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.011641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.011618 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.012831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.012807 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca" pod "1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" (UID: "1c9c0a47-af7e-4075-a0d5-9ae6767e1d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:27.012914 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.012857 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b" pod "1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" (UID: "1c9c0a47-af7e-4075-a0d5-9ae6767e1d23"). InnerVolumeSpecName "kube-api-access-nnt9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:27.013332 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.013313 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.013451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.013427 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.015038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.015020 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:27.067171 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.067139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.067377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.067358 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlb8\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.112162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.112132 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnt9b\" (UniqueName: \"kubernetes.io/projected/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-kube-api-access-nnt9b\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:27.112162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.112161 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:27.187481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.187450 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:27.394097 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:27.394063 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2ce9ba_50a2_4961_8ef8_98963de14f56.slice/crio-8785afc89155664cae3443be1c5dc000bcdc0e372b9241c69dc603f070c79921 WatchSource:0}: Error finding container 8785afc89155664cae3443be1c5dc000bcdc0e372b9241c69dc603f070c79921: Status 404 returned error can't find the container with id 8785afc89155664cae3443be1c5dc000bcdc0e372b9241c69dc603f070c79921 Nov 02 21:55:27.395802 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.395779 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm"] Nov 02 21:55:27.565811 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:27.565777 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cac4d4_2f5b_4ac4_b171_eae2bf7b6e95.slice/crio-b01e71aa29a654b96dfbf39291d1e3ea883eef61282b9a105611335a145ffd61 WatchSource:0}: Error finding container b01e71aa29a654b96dfbf39291d1e3ea883eef61282b9a105611335a145ffd61: Status 404 returned error can't find the container with id b01e71aa29a654b96dfbf39291d1e3ea883eef61282b9a105611335a145ffd61 Nov 02 21:55:27.567058 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.567028 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c"] Nov 02 21:55:27.568774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.568744 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:27.568852 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.568805 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:27.568852 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.568832 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:27.785060 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.785020 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" event={"ID":"cc2ce9ba-50a2-4961-8ef8-98963de14f56","Type":"ContainerStarted","Data":"6e6f2622cefcbb04ad43b4679e17e944d053914a75985b1f159fdb5c12de3c17"} Nov 02 21:55:27.785250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.785086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" event={"ID":"cc2ce9ba-50a2-4961-8ef8-98963de14f56","Type":"ContainerStarted","Data":"8785afc89155664cae3443be1c5dc000bcdc0e372b9241c69dc603f070c79921"} Nov 02 21:55:27.787346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.787306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" event={"ID":"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95","Type":"ContainerStarted","Data":"76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51"} Nov 02 21:55:27.787493 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.787368 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" event={"ID":"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95","Type":"ContainerStarted","Data":"b01e71aa29a654b96dfbf39291d1e3ea883eef61282b9a105611335a145ffd61"} Nov 02 21:55:27.789093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.789073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" event={"ID":"1c9c0a47-af7e-4075-a0d5-9ae6767e1d23","Type":"ContainerDied","Data":"9900295fcae098bb00bdc4fa66f1f150b4bd4a29bc107537895e8d55e33edb88"} Nov 02 21:55:27.789192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.789105 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9900295fcae098bb00bdc4fa66f1f150b4bd4a29bc107537895e8d55e33edb88" Nov 02 21:55:27.789192 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.789137 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x" Nov 02 21:55:27.898031 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:27.897979 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" podStartSLOduration=1.8979643560000001 podStartE2EDuration="1.897964356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:27.89697515 +0000 UTC m=+1275.059070270" watchObservedRunningTime="2026-11-02 21:55:27.897964356 +0000 UTC m=+1275.060059493" Nov 02 21:55:28.079419 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.079080 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podStartSLOduration=2.079060208 podStartE2EDuration="2.079060208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:28.073570243 +0000 UTC m=+1275.235665364" watchObservedRunningTime="2026-11-02 21:55:28.079060208 +0000 UTC m=+1275.241155329" Nov 02 21:55:28.194751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.189464 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:28.194751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.189595 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:28.194751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.189664 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:28.412976 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.412935 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch"] Nov 02 21:55:28.414037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.414006 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" containerName="revoke-keys" Nov 02 21:55:28.414037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.414036 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" containerName="revoke-keys" Nov 02 21:55:28.414230 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.414205 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" containerName="revoke-keys" Nov 02 21:55:28.418875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.418854 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.495274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.495233 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch"] Nov 02 21:55:28.527830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.527779 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528007 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.527866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528007 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.527885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vfc\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528007 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.527914 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528007 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.527935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528240 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.528097 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528240 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.528208 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.528241 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.528327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.528273 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.550772 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.550744 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp"] Nov 02 21:55:28.554865 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.554849 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.608626 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.608595 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp"] Nov 02 21:55:28.629257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629217 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsstx\" (UniqueName: \"kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.629444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vfc\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629366 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629663 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629663 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629606 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.629663 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629650 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629859 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629698 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.629859 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.629732 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.630508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.630508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.630508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630094 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.630508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630121 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.630508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630367 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.630838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.630838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.630808 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.632533 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.632502 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.632621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.632509 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.691600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.691552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vfc\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.691807 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.691634 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.732371 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.732297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.732490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.732387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsstx\" (UniqueName: \"kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.732606 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.732590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.732954 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.732935 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.734791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.734768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.754816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.754794 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:28.789733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.789534 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsstx\" (UniqueName: \"kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx\") pod \"maas-api-revoke-sub-e11ffbac-cf2zp\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:28.801211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.801180 5091 generic.go:356] "Generic (PLEG): container finished" podID="cc2ce9ba-50a2-4961-8ef8-98963de14f56" containerID="6e6f2622cefcbb04ad43b4679e17e944d053914a75985b1f159fdb5c12de3c17" exitCode=0 Nov 02 21:55:28.801324 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.801239 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" event={"ID":"cc2ce9ba-50a2-4961-8ef8-98963de14f56","Type":"ContainerDied","Data":"6e6f2622cefcbb04ad43b4679e17e944d053914a75985b1f159fdb5c12de3c17"} Nov 02 21:55:28.901525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:28.901339 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:29.188319 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.188282 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:29.188521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.188335 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:29.212218 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:29.212184 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411e3bef_199a_4a56_8e53_7477d83be204.slice/crio-7b0cd3f411e1ef298b547bcc46c57521e5469d391915edfc2a39457c6ca5a799 WatchSource:0}: Error finding container 7b0cd3f411e1ef298b547bcc46c57521e5469d391915edfc2a39457c6ca5a799: Status 404 returned error can't find the container with id 7b0cd3f411e1ef298b547bcc46c57521e5469d391915edfc2a39457c6ca5a799 Nov 02 21:55:29.213651 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.213628 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch"] Nov 02 21:55:29.214813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.214779 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:29.214902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.214859 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:29.214970 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.214903 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:29.341367 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:29.341327 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d26da1a_c280_409d_b4e1_25f034d00afa.slice/crio-52ca60a7201786f66b7788aa71aea19cd636299094e79c105bb35472bfcc8f3f WatchSource:0}: Error finding container 52ca60a7201786f66b7788aa71aea19cd636299094e79c105bb35472bfcc8f3f: Status 404 returned error can't find the container with id 52ca60a7201786f66b7788aa71aea19cd636299094e79c105bb35472bfcc8f3f Nov 02 21:55:29.345999 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.345934 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp"] Nov 02 21:55:29.811960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.811923 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" event={"ID":"0d26da1a-c280-409d-b4e1-25f034d00afa","Type":"ContainerStarted","Data":"52ca60a7201786f66b7788aa71aea19cd636299094e79c105bb35472bfcc8f3f"} Nov 02 21:55:29.813867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.813833 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" event={"ID":"411e3bef-199a-4a56-8e53-7477d83be204","Type":"ContainerStarted","Data":"c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1"} Nov 02 21:55:29.814010 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:29.813891 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" event={"ID":"411e3bef-199a-4a56-8e53-7477d83be204","Type":"ContainerStarted","Data":"7b0cd3f411e1ef298b547bcc46c57521e5469d391915edfc2a39457c6ca5a799"} Nov 02 21:55:30.117083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.116664 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:30.152751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.152719 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf\" (UniqueName: \"kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf\") pod \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " Nov 02 21:55:30.152931 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.152915 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca\") pod \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " Nov 02 21:55:30.152990 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.152972 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca\") pod \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\" (UID: \"cc2ce9ba-50a2-4961-8ef8-98963de14f56\") " Nov 02 21:55:30.153297 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.153275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca" pod "cc2ce9ba-50a2-4961-8ef8-98963de14f56" (UID: "cc2ce9ba-50a2-4961-8ef8-98963de14f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:30.154758 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.154735 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf" pod "cc2ce9ba-50a2-4961-8ef8-98963de14f56" (UID: "cc2ce9ba-50a2-4961-8ef8-98963de14f56"). InnerVolumeSpecName "kube-api-access-cqhjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:30.155782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.155762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca" pod "cc2ce9ba-50a2-4961-8ef8-98963de14f56" (UID: "cc2ce9ba-50a2-4961-8ef8-98963de14f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:30.199695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.192757 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:30.199695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.192819 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:30.255081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.254934 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:30.255081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.254969 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqhjf\" (UniqueName: \"kubernetes.io/projected/cc2ce9ba-50a2-4961-8ef8-98963de14f56-kube-api-access-cqhjf\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:30.255081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.254986 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ce9ba-50a2-4961-8ef8-98963de14f56-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:30.259963 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.259906 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podStartSLOduration=2.259888878 podStartE2EDuration="2.259888878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:29.947958039 +0000 UTC m=+1277.110053162" watchObservedRunningTime="2026-11-02 21:55:30.259888878 +0000 UTC m=+1277.421983998" Nov 02 21:55:30.755746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.755710 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:30.757090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.757065 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:55:30.757261 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.757111 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 21:55:30.832214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.832130 5091 generic.go:356] "Generic (PLEG): container finished" podID="0d26da1a-c280-409d-b4e1-25f034d00afa" containerID="cb95ceb6c19d648cd9dff895dfc47c0b61a143e57c38ae1332b84ea9079a7ce4" exitCode=0 Nov 02 21:55:30.832346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.832263 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" event={"ID":"0d26da1a-c280-409d-b4e1-25f034d00afa","Type":"ContainerDied","Data":"cb95ceb6c19d648cd9dff895dfc47c0b61a143e57c38ae1332b84ea9079a7ce4"} Nov 02 21:55:30.833822 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.833796 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" Nov 02 21:55:30.833937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.833842 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm" event={"ID":"cc2ce9ba-50a2-4961-8ef8-98963de14f56","Type":"ContainerDied","Data":"8785afc89155664cae3443be1c5dc000bcdc0e372b9241c69dc603f070c79921"} Nov 02 21:55:30.833937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:30.833878 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8785afc89155664cae3443be1c5dc000bcdc0e372b9241c69dc603f070c79921" Nov 02 21:55:31.198704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:31.197827 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:31.198704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:31.197911 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:31.755316 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:31.755276 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:55:31.755465 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:31.755343 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 21:55:32.124587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.123165 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr"] Nov 02 21:55:32.124587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.123845 5091 cpu_manager.go:414] "containerMap: removing container" podUID="cc2ce9ba-50a2-4961-8ef8-98963de14f56" containerName="revoke-keys" Nov 02 21:55:32.124587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.123861 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2ce9ba-50a2-4961-8ef8-98963de14f56" containerName="revoke-keys" Nov 02 21:55:32.124587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.123991 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2ce9ba-50a2-4961-8ef8-98963de14f56" containerName="revoke-keys" Nov 02 21:55:32.127876 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.127855 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.142498 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.142479 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:32.155928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.155905 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr"] Nov 02 21:55:32.220189 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.220162 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:32.220611 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.220590 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:32.229096 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.229075 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca\") pod \"0d26da1a-c280-409d-b4e1-25f034d00afa\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " Nov 02 21:55:32.229648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.229634 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx\" (UniqueName: \"kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx\") pod \"0d26da1a-c280-409d-b4e1-25f034d00afa\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " Nov 02 21:55:32.230086 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.230072 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca\") pod \"0d26da1a-c280-409d-b4e1-25f034d00afa\" (UID: \"0d26da1a-c280-409d-b4e1-25f034d00afa\") " Nov 02 21:55:32.230538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.229486 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca" pod "0d26da1a-c280-409d-b4e1-25f034d00afa" (UID: "0d26da1a-c280-409d-b4e1-25f034d00afa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:32.230777 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.230754 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.230903 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.230886 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.231075 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.231057 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfpwj\" (UniqueName: \"kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.231306 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.231289 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:32.232151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.232116 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx" pod "0d26da1a-c280-409d-b4e1-25f034d00afa" (UID: "0d26da1a-c280-409d-b4e1-25f034d00afa"). InnerVolumeSpecName "kube-api-access-zsstx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:32.232396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.232376 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca" pod "0d26da1a-c280-409d-b4e1-25f034d00afa" (UID: "0d26da1a-c280-409d-b4e1-25f034d00afa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:32.332329 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.332296 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.332488 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.332334 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.332488 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.332381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfpwj\" (UniqueName: \"kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.332661 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.332644 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsstx\" (UniqueName: \"kubernetes.io/projected/0d26da1a-c280-409d-b4e1-25f034d00afa-kube-api-access-zsstx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:32.332706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.332685 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d26da1a-c280-409d-b4e1-25f034d00afa-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:32.333108 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.333081 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.334538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.334513 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.413696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.413363 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfpwj\" (UniqueName: \"kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj\") pod \"maas-api-revoke-sub-963ace6d-k5wrr\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.447824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.447788 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:32.755421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.755335 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:55:32.755421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.755394 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 21:55:32.853758 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.853717 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" event={"ID":"0d26da1a-c280-409d-b4e1-25f034d00afa","Type":"ContainerDied","Data":"52ca60a7201786f66b7788aa71aea19cd636299094e79c105bb35472bfcc8f3f"} Nov 02 21:55:32.853902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.853774 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ca60a7201786f66b7788aa71aea19cd636299094e79c105bb35472bfcc8f3f" Nov 02 21:55:32.853902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.853838 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp" Nov 02 21:55:32.922266 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:32.921525 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr"] Nov 02 21:55:33.058822 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.058720 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s"] Nov 02 21:55:33.062551 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.062521 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453ce505-qr87s"] Nov 02 21:55:33.205695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.202356 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:33.205695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.202415 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:33.755720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.755651 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:55:33.756139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.755753 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 21:55:33.884702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.883622 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" event={"ID":"cb8b135f-96a4-48d2-8bb8-6a916863619c","Type":"ContainerStarted","Data":"d005918f1de661ca37c70ddf0430fc4e717d631367eca9405a454af190219233"} Nov 02 21:55:33.884702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:33.883717 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" event={"ID":"cb8b135f-96a4-48d2-8bb8-6a916863619c","Type":"ContainerStarted","Data":"9723814bfcffa09392ab6c5d49d7009a2cd1dc8218f90c4409ff617bcc1df707"} Nov 02 21:55:34.002282 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.002226 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" podStartSLOduration=2.002211954 podStartE2EDuration="2.002211954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:34.000017442 +0000 UTC m=+1281.162112600" watchObservedRunningTime="2026-11-02 21:55:34.002211954 +0000 UTC m=+1281.164307074" Nov 02 21:55:34.019253 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:55:34.019178 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8b135f_96a4_48d2_8bb8_6a916863619c.slice/crio-d005918f1de661ca37c70ddf0430fc4e717d631367eca9405a454af190219233.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8b135f_96a4_48d2_8bb8_6a916863619c.slice/crio-conmon-d005918f1de661ca37c70ddf0430fc4e717d631367eca9405a454af190219233.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:55:34.190885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.190222 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 21:55:34.190885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.190292 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 21:55:34.755977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.755935 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:55:34.756370 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.756008 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 21:55:34.906098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.906061 5091 generic.go:356] "Generic (PLEG): container finished" podID="cb8b135f-96a4-48d2-8bb8-6a916863619c" containerID="d005918f1de661ca37c70ddf0430fc4e717d631367eca9405a454af190219233" exitCode=0 Nov 02 21:55:34.906447 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.906333 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" event={"ID":"cb8b135f-96a4-48d2-8bb8-6a916863619c","Type":"ContainerDied","Data":"d005918f1de661ca37c70ddf0430fc4e717d631367eca9405a454af190219233"} Nov 02 21:55:34.999609 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:34.999571 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef03a9e9-aff6-4c95-ab84-882758a06b61" path="/var/lib/kubelet/pods/ef03a9e9-aff6-4c95-ab84-882758a06b61/volumes" Nov 02 21:55:35.192647 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:35.192612 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:35.193178 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:35.193153 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:35.194005 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:35.193986 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 21:55:35.763349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:35.763300 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 21:55:35.763924 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:35.763372 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 21:55:36.047984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.047886 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5"] Nov 02 21:55:36.052494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.052463 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb"] Nov 02 21:55:36.057811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.057784 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5979a010-dh7q5"] Nov 02 21:55:36.090404 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.090381 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:36.096431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.096408 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb1e7bf-kmnvb"] Nov 02 21:55:36.202480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.202448 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca\") pod \"cb8b135f-96a4-48d2-8bb8-6a916863619c\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " Nov 02 21:55:36.202886 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.202862 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj\" (UniqueName: \"kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj\") pod \"cb8b135f-96a4-48d2-8bb8-6a916863619c\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " Nov 02 21:55:36.203088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.203063 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca\") pod \"cb8b135f-96a4-48d2-8bb8-6a916863619c\" (UID: \"cb8b135f-96a4-48d2-8bb8-6a916863619c\") " Nov 02 21:55:36.203204 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.203091 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca" pod "cb8b135f-96a4-48d2-8bb8-6a916863619c" (UID: "cb8b135f-96a4-48d2-8bb8-6a916863619c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:36.208147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.203545 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:36.208147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.205804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj" pod "cb8b135f-96a4-48d2-8bb8-6a916863619c" (UID: "cb8b135f-96a4-48d2-8bb8-6a916863619c"). InnerVolumeSpecName "kube-api-access-cfpwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:36.208147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.206155 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca" pod "cb8b135f-96a4-48d2-8bb8-6a916863619c" (UID: "cb8b135f-96a4-48d2-8bb8-6a916863619c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:36.304451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.304375 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfpwj\" (UniqueName: \"kubernetes.io/projected/cb8b135f-96a4-48d2-8bb8-6a916863619c-kube-api-access-cfpwj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:36.304451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.304406 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb8b135f-96a4-48d2-8bb8-6a916863619c-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:36.777922 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.777799 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:36.778513 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.778467 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:36.779211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.779190 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 21:55:36.928273 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.928244 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" Nov 02 21:55:36.928431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.928279 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr" event={"ID":"cb8b135f-96a4-48d2-8bb8-6a916863619c","Type":"ContainerDied","Data":"9723814bfcffa09392ab6c5d49d7009a2cd1dc8218f90c4409ff617bcc1df707"} Nov 02 21:55:36.928431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.928325 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9723814bfcffa09392ab6c5d49d7009a2cd1dc8218f90c4409ff617bcc1df707" Nov 02 21:55:36.977317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.977291 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c182a9-2d44-4204-a23e-064e717739bd" path="/var/lib/kubelet/pods/02c182a9-2d44-4204-a23e-064e717739bd/volumes" Nov 02 21:55:36.977773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:36.977758 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc12f73-e0fa-4455-abe4-a60413b4f161" path="/var/lib/kubelet/pods/8cc12f73-e0fa-4455-abe4-a60413b4f161/volumes" Nov 02 21:55:37.056610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:37.056328 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj"] Nov 02 21:55:37.062764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:37.062738 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5"] Nov 02 21:55:37.067393 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:37.067371 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8684771a-kf5xj"] Nov 02 21:55:37.071242 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:37.071219 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48217aa8-thnn5"] Nov 02 21:55:37.115841 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:55:37.115802 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:55:38.122666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.122633 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp"] Nov 02 21:55:38.123243 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.123228 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0d26da1a-c280-409d-b4e1-25f034d00afa" containerName="revoke-keys" Nov 02 21:55:38.123285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.123243 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26da1a-c280-409d-b4e1-25f034d00afa" containerName="revoke-keys" Nov 02 21:55:38.123285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.123270 5091 cpu_manager.go:414] "containerMap: removing container" podUID="cb8b135f-96a4-48d2-8bb8-6a916863619c" containerName="revoke-keys" Nov 02 21:55:38.123285 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.123275 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8b135f-96a4-48d2-8bb8-6a916863619c" containerName="revoke-keys" Nov 02 21:55:38.123413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.123386 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26da1a-c280-409d-b4e1-25f034d00afa" containerName="revoke-keys" Nov 02 21:55:38.123413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.123405 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8b135f-96a4-48d2-8bb8-6a916863619c" containerName="revoke-keys" Nov 02 21:55:38.128159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.128140 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.182488 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.182464 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp"] Nov 02 21:55:38.226902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.226871 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:55:38.227080 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.226903 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:55:38.227080 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.226928 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:55:38.227485 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.227467 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:55:38.232604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.232582 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xl5s\" (UniqueName: \"kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.232698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.232638 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.232796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.232775 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.333838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.333808 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xl5s\" (UniqueName: \"kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.333947 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.333882 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.334066 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.334045 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.334772 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.334751 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.336344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.336315 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.390756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.390691 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xl5s\" (UniqueName: \"kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s\") pod \"maas-api-revoke-sub-df420285-fzmwp\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.442538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.442516 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:38.819268 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:38.819237 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2217ebb_c519_49ad_8f26_53e55ec54d8a.slice/crio-f7c0084e11086f79ccff91993730e778130b8f36ba57b3d961e21a4bd0f0b7df WatchSource:0}: Error finding container f7c0084e11086f79ccff91993730e778130b8f36ba57b3d961e21a4bd0f0b7df: Status 404 returned error can't find the container with id f7c0084e11086f79ccff91993730e778130b8f36ba57b3d961e21a4bd0f0b7df Nov 02 21:55:38.819950 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.819927 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp"] Nov 02 21:55:38.946315 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.946283 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" event={"ID":"f2217ebb-c519-49ad-8f26-53e55ec54d8a","Type":"ContainerStarted","Data":"f7c0084e11086f79ccff91993730e778130b8f36ba57b3d961e21a4bd0f0b7df"} Nov 02 21:55:38.977763 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.977735 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856f63ae-c095-4def-ae81-961ba4f44bd7" path="/var/lib/kubelet/pods/856f63ae-c095-4def-ae81-961ba4f44bd7/volumes" Nov 02 21:55:38.978220 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:38.978206 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba80a7e-6b85-4ce0-b6da-51faf3f51d1a" path="/var/lib/kubelet/pods/dba80a7e-6b85-4ce0-b6da-51faf3f51d1a/volumes" Nov 02 21:55:39.957032 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:39.957000 5091 generic.go:356] "Generic (PLEG): container finished" podID="f2217ebb-c519-49ad-8f26-53e55ec54d8a" containerID="c6544beb4cf6ab108a1721383137b4cbaf83dc5d83be5dfe70383a138d966bf1" exitCode=0 Nov 02 21:55:39.957401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:39.957050 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" event={"ID":"f2217ebb-c519-49ad-8f26-53e55ec54d8a","Type":"ContainerDied","Data":"c6544beb4cf6ab108a1721383137b4cbaf83dc5d83be5dfe70383a138d966bf1"} Nov 02 21:55:41.141965 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.141935 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:41.268389 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.268311 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s\" (UniqueName: \"kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s\") pod \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " Nov 02 21:55:41.268518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.268391 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca\") pod \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " Nov 02 21:55:41.268518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.268490 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca\") pod \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\" (UID: \"f2217ebb-c519-49ad-8f26-53e55ec54d8a\") " Nov 02 21:55:41.268820 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.268797 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca" pod "f2217ebb-c519-49ad-8f26-53e55ec54d8a" (UID: "f2217ebb-c519-49ad-8f26-53e55ec54d8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:41.269038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.269007 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:41.270756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.270724 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca" pod "f2217ebb-c519-49ad-8f26-53e55ec54d8a" (UID: "f2217ebb-c519-49ad-8f26-53e55ec54d8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:41.270846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.270760 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s" pod "f2217ebb-c519-49ad-8f26-53e55ec54d8a" (UID: "f2217ebb-c519-49ad-8f26-53e55ec54d8a"). InnerVolumeSpecName "kube-api-access-8xl5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:41.369983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.369947 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xl5s\" (UniqueName: \"kubernetes.io/projected/f2217ebb-c519-49ad-8f26-53e55ec54d8a-kube-api-access-8xl5s\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:41.369983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.369979 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2217ebb-c519-49ad-8f26-53e55ec54d8a-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:41.982924 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.982892 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" Nov 02 21:55:41.983123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.982922 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp" event={"ID":"f2217ebb-c519-49ad-8f26-53e55ec54d8a","Type":"ContainerDied","Data":"f7c0084e11086f79ccff91993730e778130b8f36ba57b3d961e21a4bd0f0b7df"} Nov 02 21:55:41.983123 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:41.982972 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7c0084e11086f79ccff91993730e778130b8f36ba57b3d961e21a4bd0f0b7df" Nov 02 21:55:43.548534 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.548501 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc"] Nov 02 21:55:43.549638 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.549613 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f2217ebb-c519-49ad-8f26-53e55ec54d8a" containerName="revoke-keys" Nov 02 21:55:43.549638 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.549637 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2217ebb-c519-49ad-8f26-53e55ec54d8a" containerName="revoke-keys" Nov 02 21:55:43.550006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.549875 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2217ebb-c519-49ad-8f26-53e55ec54d8a" containerName="revoke-keys" Nov 02 21:55:43.556065 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.555527 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.673694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.673231 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc"] Nov 02 21:55:43.703320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.703521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703350 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.703521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrjm\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.703637 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.704218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703758 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.704218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703832 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.704218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.703974 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.704218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.704001 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.704218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.704083 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.805567 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.805477 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.806072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.805958 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.806094 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.806504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.806963 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjm\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.807043 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.807196 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.807228 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.807291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.807344 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.807319 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.808436 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.808408 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.809165 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.809121 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.809350 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.809330 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.809771 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.809748 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.810110 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.810054 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.819781 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.819757 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.866692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.866393 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrjm\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.870691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.867297 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:43.879315 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:43.878392 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:44.442705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:44.430768 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc"] Nov 02 21:55:44.460960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:44.450213 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:44.460960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:44.450306 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:44.460960 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:44.450348 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:45.104155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.102951 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" event={"ID":"843bd159-e589-425e-b160-4d5e6f28e638","Type":"ContainerStarted","Data":"0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de"} Nov 02 21:55:45.104155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.103080 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" event={"ID":"843bd159-e589-425e-b160-4d5e6f28e638","Type":"ContainerStarted","Data":"52778e03e35f64b28148e483b5575a27920babb5b7ae7a28a17abf15c0e9b7c5"} Nov 02 21:55:45.234694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.224000 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podStartSLOduration=2.223979146 podStartE2EDuration="2.223979146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:45.222960474 +0000 UTC m=+1292.385055595" watchObservedRunningTime="2026-11-02 21:55:45.223979146 +0000 UTC m=+1292.386074271" Nov 02 21:55:45.877824 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.877767 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:45.878451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.878415 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 21:55:45.880079 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.880055 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:45.883209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.882879 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:45.883801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.882934 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:45.883963 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.883943 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:45.899805 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.899758 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:45.967890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:45.967822 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.069324 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.069286 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl"] Nov 02 21:55:46.077802 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.077762 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:55:46.197275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.197174 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.229799 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.207277 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603095a0-cc4hl"] Nov 02 21:55:46.791694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.790767 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.852696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.851771 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.862797 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.862752 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.889821 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.889758 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 21:55:46.890455 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.890422 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 02 21:55:46.890935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.890913 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 21:55:46.891357 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.891338 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:55:46.891509 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.891497 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:55:46.891620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.891611 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:55:46.899608 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.899583 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:46.899796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.899774 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:46.900947 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.900921 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.901398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:46.901379 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:55:47.027700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.023812 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.027700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.025310 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:55:47.146760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.131378 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:55:47.146760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.135152 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:55:47.231695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.218841 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.231695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.221813 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:47.258826 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.258760 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.776695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.721581 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded" Nov 02 21:55:47.858860 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.858801 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.871704 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.871658 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad54f272-861e-45fe-ab4b-525ce8a61713" path="/var/lib/kubelet/pods/ad54f272-861e-45fe-ab4b-525ce8a61713/volumes" Nov 02 21:55:47.895729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.895362 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:55:47.895729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.895431 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:55:47.896138 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.896113 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 21:55:47.896308 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.875824 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.896588 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.875858 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 21:55:47.897517 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.896059 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.937484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.898013 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.937484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.919982 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:55:47.937484 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.920823 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:55:47.939762 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.939719 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.958787 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.955637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:55:47.976111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.974113 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:47.976111 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:47.974170 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:48.026892 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.026832 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:48.145848 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.145802 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:55:48.222609 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.221923 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:48.342515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.315711 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 02 21:55:48.678694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.660696 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:55:48.771994 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.771958 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:55:48.920855 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.920812 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:55:48.921422 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.921393 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 21:55:48.925925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.925895 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 21:55:48.926369 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.926344 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 21:55:48.937131 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.937053 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 02 21:55:49.007928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.007799 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:49.007928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.007866 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:49.039547 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.029760 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.242777 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.213263 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:55:49.528753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:48.953177 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.528753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.519571 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:55:49.528753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.520389 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:55:49.553687 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.552239 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl"] Nov 02 21:55:49.836320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.836236 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl"] Nov 02 21:55:49.836693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.836653 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:49.860157 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.859758 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:49.860157 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.859834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:49.860157 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.859877 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnk7w\" (UniqueName: \"kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:49.896337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.895291 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:49.896337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.895372 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:49.897077 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.897053 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:55:49.897396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.897378 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:55:49.897627 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.897611 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:55:49.897901 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.897883 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:55:49.935866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.935801 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 21:55:49.936548 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:49.936511 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:55:50.020697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.005418 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.020697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.005500 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.020697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.005535 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnk7w\" (UniqueName: \"kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.037391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.021860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.037391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.028517 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:55:50.037391 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.028574 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:55:50.056427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.056375 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:55:50.081694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.075742 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d"] Nov 02 21:55:50.128742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.109108 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.128742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.121086 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.128742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.123355 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:55:50.131137 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.131105 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnk7w\" (UniqueName: \"kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w\") pod \"maas-api-revoke-sub-12cf4174-mshtl\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.135899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.135833 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d"] Nov 02 21:55:50.143491 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.142986 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4"] Nov 02 21:55:50.147424 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.147403 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcd6548-jpfd4"] Nov 02 21:55:50.213160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.213136 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:50.309946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.309915 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l6f\" (UniqueName: \"kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.310122 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.310107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.310170 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.310157 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.411581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.411484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.411581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.411558 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.411829 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.411649 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l6f\" (UniqueName: \"kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.412411 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.412380 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.414034 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.414008 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.493694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.493613 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l6f\" (UniqueName: \"kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f\") pod \"maas-api-revoke-sub-57bfadc4-9295d\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.629368 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:50.629334 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bd8392_45fe_4c9b_beb8_e80861302be6.slice/crio-f85eac19fc786f395867b3c7c1f987c9fd9dd555fdedd95274f31acd891157f9 WatchSource:0}: Error finding container f85eac19fc786f395867b3c7c1f987c9fd9dd555fdedd95274f31acd891157f9: Status 404 returned error can't find the container with id f85eac19fc786f395867b3c7c1f987c9fd9dd555fdedd95274f31acd891157f9 Nov 02 21:55:50.629799 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.629778 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl"] Nov 02 21:55:50.776700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.775603 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:50.884703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.884015 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:50.884703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:50.884070 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:51.000832 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.000796 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466a7651-8cbf-46a0-a236-8a6e25c09522" path="/var/lib/kubelet/pods/466a7651-8cbf-46a0-a236-8a6e25c09522/volumes" Nov 02 21:55:51.100388 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.100349 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" event={"ID":"f1bd8392-45fe-4c9b-beb8-e80861302be6","Type":"ContainerStarted","Data":"f85eac19fc786f395867b3c7c1f987c9fd9dd555fdedd95274f31acd891157f9"} Nov 02 21:55:51.260658 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:51.260620 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f38015d_ed9e_42fd_b4c1_4c6cbebcc26f.slice/crio-305d589d46adf3e88aaeb90edf57176dfc6aef1b58946b4ac3b0a52b1ff0ca9a WatchSource:0}: Error finding container 305d589d46adf3e88aaeb90edf57176dfc6aef1b58946b4ac3b0a52b1ff0ca9a: Status 404 returned error can't find the container with id 305d589d46adf3e88aaeb90edf57176dfc6aef1b58946b4ac3b0a52b1ff0ca9a Nov 02 21:55:51.261596 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.261561 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d"] Nov 02 21:55:51.890396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.890350 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:51.890935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.890416 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:51.921946 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.921915 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848"] Nov 02 21:55:51.928088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.928067 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:51.933795 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.933762 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:51.933932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.933852 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:51.933932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:51.933885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl775\" (UniqueName: \"kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.003827 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.003791 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848"] Nov 02 21:55:52.035383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.035302 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.035383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.035373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.035604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.035399 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl775\" (UniqueName: \"kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.036212 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.036183 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.037804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.037779 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.045250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.045225 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9"] Nov 02 21:55:52.049875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.049854 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2249c8-6fxt9"] Nov 02 21:55:52.092475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.092340 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl775\" (UniqueName: \"kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775\") pod \"maas-api-revoke-sub-a07b1a83-pb848\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.110198 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.110162 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" event={"ID":"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f","Type":"ContainerStarted","Data":"f748bc15d79521557e46125d270f33fde352fb275beab591b9f29fa64f027e56"} Nov 02 21:55:52.110349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.110219 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" event={"ID":"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f","Type":"ContainerStarted","Data":"305d589d46adf3e88aaeb90edf57176dfc6aef1b58946b4ac3b0a52b1ff0ca9a"} Nov 02 21:55:52.112127 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.112097 5091 generic.go:356] "Generic (PLEG): container finished" podID="f1bd8392-45fe-4c9b-beb8-e80861302be6" containerID="e22abb510a975df72d739ba6736144464617b7f1597a499ace23532a9fa8811d" exitCode=0 Nov 02 21:55:52.112218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.112169 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" event={"ID":"f1bd8392-45fe-4c9b-beb8-e80861302be6","Type":"ContainerDied","Data":"e22abb510a975df72d739ba6736144464617b7f1597a499ace23532a9fa8811d"} Nov 02 21:55:52.224818 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.224754 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" podStartSLOduration=3.224741324 podStartE2EDuration="3.224741324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:52.220711236 +0000 UTC m=+1299.382806356" watchObservedRunningTime="2026-11-02 21:55:52.224741324 +0000 UTC m=+1299.386836443" Nov 02 21:55:52.244526 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.244500 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:52.700899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.700373 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848"] Nov 02 21:55:52.886796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.882091 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:52.886796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:52.882169 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:53.000230 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.000155 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6476bc3f-9806-4f7e-8a00-5f4e53f2562e" path="/var/lib/kubelet/pods/6476bc3f-9806-4f7e-8a00-5f4e53f2562e/volumes" Nov 02 21:55:53.123088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.123057 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" event={"ID":"fbf6f582-d150-497a-bdcb-f2a4a528786d","Type":"ContainerStarted","Data":"61a3a3ba254b9c06230392a5099323d3793bf13e8641819100568188af34daf2"} Nov 02 21:55:53.123221 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.123113 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" event={"ID":"fbf6f582-d150-497a-bdcb-f2a4a528786d","Type":"ContainerStarted","Data":"2cbf5d08a55f65d6c7a19afe5bbbf883d540e4115ba347ea87f51abe5fb380cc"} Nov 02 21:55:53.124999 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.124975 5091 generic.go:356] "Generic (PLEG): container finished" podID="8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" containerID="f748bc15d79521557e46125d270f33fde352fb275beab591b9f29fa64f027e56" exitCode=0 Nov 02 21:55:53.125134 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.125038 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" event={"ID":"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f","Type":"ContainerDied","Data":"f748bc15d79521557e46125d270f33fde352fb275beab591b9f29fa64f027e56"} Nov 02 21:55:53.240169 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.240110 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" podStartSLOduration=2.240090995 podStartE2EDuration="2.240090995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:53.237616704 +0000 UTC m=+1300.399711835" watchObservedRunningTime="2026-11-02 21:55:53.240090995 +0000 UTC m=+1300.402186116" Nov 02 21:55:53.347651 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.347626 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:53.453460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.453423 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w\" (UniqueName: \"kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w\") pod \"f1bd8392-45fe-4c9b-beb8-e80861302be6\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " Nov 02 21:55:53.453620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.453467 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca\") pod \"f1bd8392-45fe-4c9b-beb8-e80861302be6\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " Nov 02 21:55:53.453620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.453512 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca\") pod \"f1bd8392-45fe-4c9b-beb8-e80861302be6\" (UID: \"f1bd8392-45fe-4c9b-beb8-e80861302be6\") " Nov 02 21:55:53.454167 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.454130 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca" pod "f1bd8392-45fe-4c9b-beb8-e80861302be6" (UID: "f1bd8392-45fe-4c9b-beb8-e80861302be6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:53.455698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.455655 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w" pod "f1bd8392-45fe-4c9b-beb8-e80861302be6" (UID: "f1bd8392-45fe-4c9b-beb8-e80861302be6"). InnerVolumeSpecName "kube-api-access-hnk7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:53.455813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.455738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca" pod "f1bd8392-45fe-4c9b-beb8-e80861302be6" (UID: "f1bd8392-45fe-4c9b-beb8-e80861302be6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:53.555051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.555017 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnk7w\" (UniqueName: \"kubernetes.io/projected/f1bd8392-45fe-4c9b-beb8-e80861302be6-kube-api-access-hnk7w\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:53.555051 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.555051 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:53.555271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.555066 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bd8392-45fe-4c9b-beb8-e80861302be6-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:53.888151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.888067 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:53.888151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:53.888138 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:54.047866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.047783 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q"] Nov 02 21:55:54.052416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.052386 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b67a2a3-kr95q"] Nov 02 21:55:54.137330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.137284 5091 generic.go:356] "Generic (PLEG): container finished" podID="fbf6f582-d150-497a-bdcb-f2a4a528786d" containerID="61a3a3ba254b9c06230392a5099323d3793bf13e8641819100568188af34daf2" exitCode=0 Nov 02 21:55:54.137522 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.137369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" event={"ID":"fbf6f582-d150-497a-bdcb-f2a4a528786d","Type":"ContainerDied","Data":"61a3a3ba254b9c06230392a5099323d3793bf13e8641819100568188af34daf2"} Nov 02 21:55:54.139222 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.139198 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" Nov 02 21:55:54.139353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.139228 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl" event={"ID":"f1bd8392-45fe-4c9b-beb8-e80861302be6","Type":"ContainerDied","Data":"f85eac19fc786f395867b3c7c1f987c9fd9dd555fdedd95274f31acd891157f9"} Nov 02 21:55:54.139353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.139276 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85eac19fc786f395867b3c7c1f987c9fd9dd555fdedd95274f31acd891157f9" Nov 02 21:55:54.354508 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.354474 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:54.364268 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.364230 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f\" (UniqueName: \"kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f\") pod \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " Nov 02 21:55:54.364435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.364403 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca\") pod \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " Nov 02 21:55:54.364509 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.364452 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca\") pod \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\" (UID: \"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f\") " Nov 02 21:55:54.365233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.365206 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca" pod "8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" (UID: "8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:54.366549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.366524 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca" pod "8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" (UID: "8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:54.366811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.366787 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f" pod "8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" (UID: "8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f"). InnerVolumeSpecName "kube-api-access-29l6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:54.466399 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.466362 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:54.466399 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.466401 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:54.466620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.466417 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29l6f\" (UniqueName: \"kubernetes.io/projected/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f-kube-api-access-29l6f\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:54.878811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.878776 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:55:54.878977 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.878837 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 21:55:54.978649 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:54.978618 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ba28f6-cd88-4006-9f20-21207be10eab" path="/var/lib/kubelet/pods/24ba28f6-cd88-4006-9f20-21207be10eab/volumes" Nov 02 21:55:55.153630 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.153533 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" event={"ID":"8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f","Type":"ContainerDied","Data":"305d589d46adf3e88aaeb90edf57176dfc6aef1b58946b4ac3b0a52b1ff0ca9a"} Nov 02 21:55:55.153630 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.153561 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d" Nov 02 21:55:55.153630 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.153591 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="305d589d46adf3e88aaeb90edf57176dfc6aef1b58946b4ac3b0a52b1ff0ca9a" Nov 02 21:55:55.326573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.326545 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:55.379101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.379063 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775\" (UniqueName: \"kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775\") pod \"fbf6f582-d150-497a-bdcb-f2a4a528786d\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " Nov 02 21:55:55.379262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.379129 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca\") pod \"fbf6f582-d150-497a-bdcb-f2a4a528786d\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " Nov 02 21:55:55.379338 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.379292 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca\") pod \"fbf6f582-d150-497a-bdcb-f2a4a528786d\" (UID: \"fbf6f582-d150-497a-bdcb-f2a4a528786d\") " Nov 02 21:55:55.379738 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.379708 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca" pod "fbf6f582-d150-497a-bdcb-f2a4a528786d" (UID: "fbf6f582-d150-497a-bdcb-f2a4a528786d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:55.379903 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.379883 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:55.382102 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.382061 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775" pod "fbf6f582-d150-497a-bdcb-f2a4a528786d" (UID: "fbf6f582-d150-497a-bdcb-f2a4a528786d"). InnerVolumeSpecName "kube-api-access-kl775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:55.383782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.383751 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca" pod "fbf6f582-d150-497a-bdcb-f2a4a528786d" (UID: "fbf6f582-d150-497a-bdcb-f2a4a528786d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:55.482251 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.482165 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl775\" (UniqueName: \"kubernetes.io/projected/fbf6f582-d150-497a-bdcb-f2a4a528786d-kube-api-access-kl775\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:55.482251 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.482207 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf6f582-d150-497a-bdcb-f2a4a528786d-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:55.890459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.890421 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:55.890866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.890843 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:55.891457 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:55.891440 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 21:55:56.166883 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:56.166808 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" Nov 02 21:55:56.166883 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:56.166841 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848" event={"ID":"fbf6f582-d150-497a-bdcb-f2a4a528786d","Type":"ContainerDied","Data":"2cbf5d08a55f65d6c7a19afe5bbbf883d540e4115ba347ea87f51abe5fb380cc"} Nov 02 21:55:56.167399 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:56.166933 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cbf5d08a55f65d6c7a19afe5bbbf883d540e4115ba347ea87f51abe5fb380cc" Nov 02 21:55:57.613631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.613594 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm"] Nov 02 21:55:57.614275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614254 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fbf6f582-d150-497a-bdcb-f2a4a528786d" containerName="revoke-keys" Nov 02 21:55:57.614275 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614273 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf6f582-d150-497a-bdcb-f2a4a528786d" containerName="revoke-keys" Nov 02 21:55:57.614439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614308 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f1bd8392-45fe-4c9b-beb8-e80861302be6" containerName="revoke-keys" Nov 02 21:55:57.614439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614313 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bd8392-45fe-4c9b-beb8-e80861302be6" containerName="revoke-keys" Nov 02 21:55:57.614439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614328 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" containerName="revoke-keys" Nov 02 21:55:57.614439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614336 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" containerName="revoke-keys" Nov 02 21:55:57.614566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614468 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf6f582-d150-497a-bdcb-f2a4a528786d" containerName="revoke-keys" Nov 02 21:55:57.614566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614480 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bd8392-45fe-4c9b-beb8-e80861302be6" containerName="revoke-keys" Nov 02 21:55:57.614566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.614488 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" containerName="revoke-keys" Nov 02 21:55:57.618570 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.618549 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.674796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.674769 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm"] Nov 02 21:55:57.710899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.710861 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.711073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.710908 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.711073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.711048 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pfv\" (UniqueName: \"kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.717172 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.717151 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:55:57.717283 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.717202 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:55:57.719607 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.719589 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:55:57.719660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.719589 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:55:57.812917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.812877 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.813092 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.812929 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.813092 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.812980 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pfv\" (UniqueName: \"kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.813610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.813588 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.815290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.815270 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.869395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.869369 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pfv\" (UniqueName: \"kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv\") pod \"maas-api-revoke-sub-0bf00d86-bgrrm\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:57.932916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:57.932892 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:55:58.080022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.079984 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2"] Nov 02 21:55:58.086735 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.086713 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.116906 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.116880 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117018 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.116940 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcrv\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117062 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117095 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117064 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117133 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117095 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117168 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117263 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117276 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.117301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.117294 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.169186 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.169105 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2"] Nov 02 21:55:58.218803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.218773 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.218803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.218805 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219028 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.218899 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219028 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.218978 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcrv\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219036 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219064 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219147 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219114 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219147 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219466 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219582 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219661 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.219764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.219711 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.221396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.221376 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.221644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.221627 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.275602 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.275572 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.275755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.275660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcrv\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.308781 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:58.308749 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf24783e_99da_4187_a9bc_902adb087a9b.slice/crio-ac90ca436d2d344de0d6670856c13b54ac7264cb6f057122ca08365ba2711d9b WatchSource:0}: Error finding container ac90ca436d2d344de0d6670856c13b54ac7264cb6f057122ca08365ba2711d9b: Status 404 returned error can't find the container with id ac90ca436d2d344de0d6670856c13b54ac7264cb6f057122ca08365ba2711d9b Nov 02 21:55:58.310567 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.310542 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm"] Nov 02 21:55:58.402035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.401964 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:58.687489 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.687457 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x"] Nov 02 21:55:58.691795 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.691779 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.728489 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.728425 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.728590 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.728538 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfxw\" (UniqueName: \"kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.728662 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.728647 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.749154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.749122 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x"] Nov 02 21:55:58.835599 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:55:58.835561 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda17aa08_9009_40bc_8576_643e3d832fe4.slice/crio-5d7c569ff55a2ef3e429a2e79176df0c0a4f6a79fd49f9339d9d206dfd2457ec WatchSource:0}: Error finding container 5d7c569ff55a2ef3e429a2e79176df0c0a4f6a79fd49f9339d9d206dfd2457ec: Status 404 returned error can't find the container with id 5d7c569ff55a2ef3e429a2e79176df0c0a4f6a79fd49f9339d9d206dfd2457ec Nov 02 21:55:58.837618 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.837594 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.838019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.837995 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.838130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.838082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfxw\" (UniqueName: \"kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.839112 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.839089 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.839725 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.839698 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2"] Nov 02 21:55:58.839846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.839711 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:58.839846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.839811 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:58.839846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.839859 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:58.842135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.840589 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:58.897624 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:58.897600 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfxw\" (UniqueName: \"kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw\") pod \"maas-api-revoke-sub-3325cdd9-xg85x\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:59.005932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.005601 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:55:59.018101 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.018068 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99"] Nov 02 21:55:59.023545 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.023526 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.081504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.081473 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-96c052-openshift-default-dockercfg-fh882\"" Nov 02 21:55:59.098930 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.096592 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99"] Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.140896 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.140959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.140999 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.141071 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.141094 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.141138 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.141272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.141327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.141682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.141352 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ws88\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.199754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.199713 5091 generic.go:356] "Generic (PLEG): container finished" podID="bf24783e-99da-4187-a9bc-902adb087a9b" containerID="2d78c1d8785ed405bddf4fa09cdf2c03fdbb7d58028b60e0cbb959c041ff41fa" exitCode=0 Nov 02 21:55:59.199930 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.199837 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" event={"ID":"bf24783e-99da-4187-a9bc-902adb087a9b","Type":"ContainerDied","Data":"2d78c1d8785ed405bddf4fa09cdf2c03fdbb7d58028b60e0cbb959c041ff41fa"} Nov 02 21:55:59.199930 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.199899 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" event={"ID":"bf24783e-99da-4187-a9bc-902adb087a9b","Type":"ContainerStarted","Data":"ac90ca436d2d344de0d6670856c13b54ac7264cb6f057122ca08365ba2711d9b"} Nov 02 21:55:59.201793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.201767 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" event={"ID":"da17aa08-9009-40bc-8576-643e3d832fe4","Type":"ContainerStarted","Data":"c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4"} Nov 02 21:55:59.201904 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.201811 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" event={"ID":"da17aa08-9009-40bc-8576-643e3d832fe4","Type":"ContainerStarted","Data":"5d7c569ff55a2ef3e429a2e79176df0c0a4f6a79fd49f9339d9d206dfd2457ec"} Nov 02 21:55:59.242177 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242149 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242202 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ws88\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242327 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242508 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242852 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242831 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.242932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.243002 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.242978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.243171 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.243149 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.243246 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.243221 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.244829 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.244798 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.244923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.244804 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.298305 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.298234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ws88\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.298437 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.298362 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.346602 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.346537 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:55:59.363030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.363003 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7196-e0c15f"] Nov 02 21:55:59.368629 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.368568 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:55:59.404701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.402594 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x"] Nov 02 21:55:59.404701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.402699 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:55:59.410427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.409648 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:55:59.410427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.409776 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:55:59.420918 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.420827 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m65jt\"" Nov 02 21:55:59.428948 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.428920 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7196-e0c15f"] Nov 02 21:55:59.441007 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.440955 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podStartSLOduration=1.440937686 podStartE2EDuration="1.440937686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:59.429509878 +0000 UTC m=+1306.591604998" watchObservedRunningTime="2026-11-02 21:55:59.440937686 +0000 UTC m=+1306.603032808" Nov 02 21:55:59.445870 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.445821 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcqp\" (UniqueName: \"kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp\") pod \"test-curl-7196-e0c15f\" (UID: \"571c31cd-de35-4790-9494-f9b869ff0904\") " pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:55:59.547191 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.547152 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcqp\" (UniqueName: \"kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp\") pod \"test-curl-7196-e0c15f\" (UID: \"571c31cd-de35-4790-9494-f9b869ff0904\") " pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:55:59.603154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.603075 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcqp\" (UniqueName: \"kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp\") pod \"test-curl-7196-e0c15f\" (UID: \"571c31cd-de35-4790-9494-f9b869ff0904\") " pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:55:59.690741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.689874 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:55:59.750076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.750046 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99"] Nov 02 21:55:59.893629 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.893579 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:59.893774 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.893732 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:59.893853 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:55:59.893777 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:00.159003 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.158970 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7196-e0c15f"] Nov 02 21:56:00.181693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.178186 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:56:00.213576 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.213535 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" event={"ID":"59cc358c-2ab2-4429-a67e-7e13adce3d60","Type":"ContainerStarted","Data":"3c08e52c45f39398e4bdae59dd5f5114f36fdadffd63a0e58fb04c4e8d1f4f38"} Nov 02 21:56:00.213741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.213624 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" event={"ID":"59cc358c-2ab2-4429-a67e-7e13adce3d60","Type":"ContainerStarted","Data":"4fa285a8fc0499fdb0d0f8a3ade08ad750d28f8d3ee5f122c024d9e668f3b04a"} Nov 02 21:56:00.214984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.214958 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" event={"ID":"05ec544d-1961-4f2a-a6e3-3d6122765837","Type":"ContainerStarted","Data":"cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b"} Nov 02 21:56:00.215066 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.215006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" event={"ID":"05ec544d-1961-4f2a-a6e3-3d6122765837","Type":"ContainerStarted","Data":"8c6ee7d5862ef62cc4e176c5004ce5755318414b6f4917e6974e19badd06957f"} Nov 02 21:56:00.216477 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.216456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7196-e0c15f" event={"ID":"571c31cd-de35-4790-9494-f9b869ff0904","Type":"ContainerStarted","Data":"82c5c996d7d6ac4c351abe679c230a45182975143e426f1c620a257d9d9b7c99"} Nov 02 21:56:00.332410 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.332354 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" podStartSLOduration=2.332340844 podStartE2EDuration="2.332340844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:00.329561772 +0000 UTC m=+1307.491656893" watchObservedRunningTime="2026-11-02 21:56:00.332340844 +0000 UTC m=+1307.494435964" Nov 02 21:56:00.347791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.347638 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:56:00.349454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.349411 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:00.349595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.349468 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:00.403351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.403300 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:56:00.403536 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.403374 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:56:00.446187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.446162 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:56:00.525017 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.521852 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podStartSLOduration=2.5218319300000003 podStartE2EDuration="2.52183193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:00.479632479 +0000 UTC m=+1307.641727600" watchObservedRunningTime="2026-11-02 21:56:00.52183193 +0000 UTC m=+1307.683927096" Nov 02 21:56:00.587502 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.587459 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv\" (UniqueName: \"kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv\") pod \"bf24783e-99da-4187-a9bc-902adb087a9b\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " Nov 02 21:56:00.588315 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.588288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca\") pod \"bf24783e-99da-4187-a9bc-902adb087a9b\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " Nov 02 21:56:00.594257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.594206 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca" pod "bf24783e-99da-4187-a9bc-902adb087a9b" (UID: "bf24783e-99da-4187-a9bc-902adb087a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:00.594463 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.594446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca\") pod \"bf24783e-99da-4187-a9bc-902adb087a9b\" (UID: \"bf24783e-99da-4187-a9bc-902adb087a9b\") " Nov 02 21:56:00.600838 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.600813 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:00.624094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.624034 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv" pod "bf24783e-99da-4187-a9bc-902adb087a9b" (UID: "bf24783e-99da-4187-a9bc-902adb087a9b"). InnerVolumeSpecName "kube-api-access-46pfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:00.626466 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.626428 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca" pod "bf24783e-99da-4187-a9bc-902adb087a9b" (UID: "bf24783e-99da-4187-a9bc-902adb087a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:00.714907 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.714798 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf24783e-99da-4187-a9bc-902adb087a9b-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:00.714907 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:00.714844 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pfv\" (UniqueName: \"kubernetes.io/projected/bf24783e-99da-4187-a9bc-902adb087a9b-kube-api-access-46pfv\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:01.048334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.048296 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx"] Nov 02 21:56:01.060228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.060198 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adbb28f-sp5vx"] Nov 02 21:56:01.239019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.238983 5091 generic.go:356] "Generic (PLEG): container finished" podID="59cc358c-2ab2-4429-a67e-7e13adce3d60" containerID="3c08e52c45f39398e4bdae59dd5f5114f36fdadffd63a0e58fb04c4e8d1f4f38" exitCode=0 Nov 02 21:56:01.239211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.239118 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" event={"ID":"59cc358c-2ab2-4429-a67e-7e13adce3d60","Type":"ContainerDied","Data":"3c08e52c45f39398e4bdae59dd5f5114f36fdadffd63a0e58fb04c4e8d1f4f38"} Nov 02 21:56:01.240985 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.240965 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" Nov 02 21:56:01.240985 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.240978 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm" event={"ID":"bf24783e-99da-4187-a9bc-902adb087a9b","Type":"ContainerDied","Data":"ac90ca436d2d344de0d6670856c13b54ac7264cb6f057122ca08365ba2711d9b"} Nov 02 21:56:01.241167 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.241003 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac90ca436d2d344de0d6670856c13b54ac7264cb6f057122ca08365ba2711d9b" Nov 02 21:56:01.347565 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.347488 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:01.347818 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.347794 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:01.419207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.419034 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:56:01.419207 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:01.419092 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:56:02.050112 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.050062 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt"] Nov 02 21:56:02.056055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.056025 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97389dd-d2fnt"] Nov 02 21:56:02.348692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.347302 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:02.348692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.347357 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:02.408695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.407656 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:56:02.408695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.407728 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:56:02.617880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.617850 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:56:02.662338 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.662300 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca\") pod \"59cc358c-2ab2-4429-a67e-7e13adce3d60\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " Nov 02 21:56:02.662338 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.662339 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca\") pod \"59cc358c-2ab2-4429-a67e-7e13adce3d60\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " Nov 02 21:56:02.662568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.662389 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw\" (UniqueName: \"kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw\") pod \"59cc358c-2ab2-4429-a67e-7e13adce3d60\" (UID: \"59cc358c-2ab2-4429-a67e-7e13adce3d60\") " Nov 02 21:56:02.720713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.716910 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca" pod "59cc358c-2ab2-4429-a67e-7e13adce3d60" (UID: "59cc358c-2ab2-4429-a67e-7e13adce3d60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:02.720713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.718046 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw" pod "59cc358c-2ab2-4429-a67e-7e13adce3d60" (UID: "59cc358c-2ab2-4429-a67e-7e13adce3d60"). InnerVolumeSpecName "kube-api-access-dlfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:02.720713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.718528 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca" pod "59cc358c-2ab2-4429-a67e-7e13adce3d60" (UID: "59cc358c-2ab2-4429-a67e-7e13adce3d60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:02.724707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.723774 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8"] Nov 02 21:56:02.724707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.724476 5091 cpu_manager.go:414] "containerMap: removing container" podUID="59cc358c-2ab2-4429-a67e-7e13adce3d60" containerName="revoke-keys" Nov 02 21:56:02.724707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.724493 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cc358c-2ab2-4429-a67e-7e13adce3d60" containerName="revoke-keys" Nov 02 21:56:02.724707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.724547 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bf24783e-99da-4187-a9bc-902adb087a9b" containerName="revoke-keys" Nov 02 21:56:02.724707 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.724555 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf24783e-99da-4187-a9bc-902adb087a9b" containerName="revoke-keys" Nov 02 21:56:02.725055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.724735 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf24783e-99da-4187-a9bc-902adb087a9b" containerName="revoke-keys" Nov 02 21:56:02.725055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.724757 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cc358c-2ab2-4429-a67e-7e13adce3d60" containerName="revoke-keys" Nov 02 21:56:02.729398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.729292 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.732868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.732844 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8"] Nov 02 21:56:02.798694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.797734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.798694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.797852 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.798694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.797977 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxs7q\" (UniqueName: \"kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.798694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.798094 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:02.798694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.798114 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59cc358c-2ab2-4429-a67e-7e13adce3d60-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:02.798694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.798131 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfxw\" (UniqueName: \"kubernetes.io/projected/59cc358c-2ab2-4429-a67e-7e13adce3d60-kube-api-access-dlfxw\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:02.909744 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.906162 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxs7q\" (UniqueName: \"kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.919624 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.918861 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.919624 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.919016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.919990 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.919955 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:02.923417 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:02.923389 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:03.014716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.010803 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxs7q\" (UniqueName: \"kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q\") pod \"maas-api-revoke-sub-e9d16649-c2jb8\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:03.026081 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.026045 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2" path="/var/lib/kubelet/pods/0bcbdbb3-4a8d-45bc-b0a1-f3ff5ae536e2/volumes" Nov 02 21:56:03.026593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.026576 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3a4ffc-6a33-4bf4-aaca-0944458a9084" path="/var/lib/kubelet/pods/bb3a4ffc-6a33-4bf4-aaca-0944458a9084/volumes" Nov 02 21:56:03.057155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.057086 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:03.321521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.321358 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" event={"ID":"59cc358c-2ab2-4429-a67e-7e13adce3d60","Type":"ContainerDied","Data":"4fa285a8fc0499fdb0d0f8a3ade08ad750d28f8d3ee5f122c024d9e668f3b04a"} Nov 02 21:56:03.321521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.321418 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa285a8fc0499fdb0d0f8a3ade08ad750d28f8d3ee5f122c024d9e668f3b04a" Nov 02 21:56:03.322937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.321788 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x" Nov 02 21:56:03.324595 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.324524 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v"] Nov 02 21:56:03.333695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.333324 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.347072 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.347044 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:03.347174 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.347099 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:03.413729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.412495 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:56:03.413729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.412550 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:56:03.431891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.429695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.431891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.429782 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrzm\" (UniqueName: \"kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.431891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.430007 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.434458 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.434432 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v"] Nov 02 21:56:03.525161 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:03.525117 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc9149b_5525_402f_b190_686b259d0975.slice/crio-fe62b0c563bc041e1f9debf7cfdc2488cbbc87787361b648e3f7b4f28e6065e8 WatchSource:0}: Error finding container fe62b0c563bc041e1f9debf7cfdc2488cbbc87787361b648e3f7b4f28e6065e8: Status 404 returned error can't find the container with id fe62b0c563bc041e1f9debf7cfdc2488cbbc87787361b648e3f7b4f28e6065e8 Nov 02 21:56:03.531558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.530838 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.531558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.530939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrzm\" (UniqueName: \"kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.531558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.531095 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.531558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.531383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.531983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.531707 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8"] Nov 02 21:56:03.533902 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.533627 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.638775 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.638742 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrzm\" (UniqueName: \"kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm\") pod \"maas-api-revoke-sub-a5084719-8lk5v\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:03.650912 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:03.650874 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:04.231961 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.231926 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v"] Nov 02 21:56:04.232300 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:04.232268 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963a778f_929d_4353_9087_75f7e9cb1058.slice/crio-54da9cf95b3463340df3c6b4463c62fb138455fa938bdea9d0a207a7b4abd3ae WatchSource:0}: Error finding container 54da9cf95b3463340df3c6b4463c62fb138455fa938bdea9d0a207a7b4abd3ae: Status 404 returned error can't find the container with id 54da9cf95b3463340df3c6b4463c62fb138455fa938bdea9d0a207a7b4abd3ae Nov 02 21:56:04.332866 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.332830 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" event={"ID":"5cc9149b-5525-402f-b190-686b259d0975","Type":"ContainerStarted","Data":"6ce8452c010d4bc30cbb1ef83e0a3e8b819080d5b5b46b0f75f025981263ebec"} Nov 02 21:56:04.333037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.332881 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" event={"ID":"5cc9149b-5525-402f-b190-686b259d0975","Type":"ContainerStarted","Data":"fe62b0c563bc041e1f9debf7cfdc2488cbbc87787361b648e3f7b4f28e6065e8"} Nov 02 21:56:04.334652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.334625 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" event={"ID":"963a778f-929d-4353-9087-75f7e9cb1058","Type":"ContainerStarted","Data":"54da9cf95b3463340df3c6b4463c62fb138455fa938bdea9d0a207a7b4abd3ae"} Nov 02 21:56:04.347078 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.347051 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:04.347197 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.347091 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:04.418749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.418058 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:56:04.418749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.418131 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:56:04.517999 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:04.517395 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" podStartSLOduration=2.517380362 podStartE2EDuration="2.517380362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:04.508343093 +0000 UTC m=+1311.670438215" watchObservedRunningTime="2026-11-02 21:56:04.517380362 +0000 UTC m=+1311.679475483" Nov 02 21:56:05.345037 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.345002 5091 generic.go:356] "Generic (PLEG): container finished" podID="5cc9149b-5525-402f-b190-686b259d0975" containerID="6ce8452c010d4bc30cbb1ef83e0a3e8b819080d5b5b46b0f75f025981263ebec" exitCode=0 Nov 02 21:56:05.345444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.345092 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" event={"ID":"5cc9149b-5525-402f-b190-686b259d0975","Type":"ContainerDied","Data":"6ce8452c010d4bc30cbb1ef83e0a3e8b819080d5b5b46b0f75f025981263ebec"} Nov 02 21:56:05.346869 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.346842 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:05.347002 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.346901 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:05.347872 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.347843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" event={"ID":"963a778f-929d-4353-9087-75f7e9cb1058","Type":"ContainerStarted","Data":"c1aaf0d1d3f1f9f682a0e29d6573b09f12e34ddbf7fb0cc0ae78318cd8df5efd"} Nov 02 21:56:05.403166 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.403133 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 21:56:05.403358 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.403181 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 21:56:05.592701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:05.573774 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" podStartSLOduration=2.573754698 podStartE2EDuration="2.573754698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:05.571080016 +0000 UTC m=+1312.733175139" watchObservedRunningTime="2026-11-02 21:56:05.573754698 +0000 UTC m=+1312.735849819" Nov 02 21:56:06.347130 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.347092 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:56:06.347718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.347661 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 21:56:06.362327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.362291 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7196-e0c15f" event={"ID":"571c31cd-de35-4790-9494-f9b869ff0904","Type":"ContainerStarted","Data":"765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712"} Nov 02 21:56:06.364478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.364450 5091 generic.go:356] "Generic (PLEG): container finished" podID="963a778f-929d-4353-9087-75f7e9cb1058" containerID="c1aaf0d1d3f1f9f682a0e29d6573b09f12e34ddbf7fb0cc0ae78318cd8df5efd" exitCode=0 Nov 02 21:56:06.364591 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.364528 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" event={"ID":"963a778f-929d-4353-9087-75f7e9cb1058","Type":"ContainerDied","Data":"c1aaf0d1d3f1f9f682a0e29d6573b09f12e34ddbf7fb0cc0ae78318cd8df5efd"} Nov 02 21:56:06.407031 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.406808 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:56:06.407634 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.407507 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:56:06.408887 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.408473 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 21:56:06.480691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.480533 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7196-e0c15f" podStartSLOduration=2.40715986 podStartE2EDuration="7.480512173s" totalImagesPullingTime="5.073352313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:06.47324669 +0000 UTC m=+1313.635341810" watchObservedRunningTime="2026-11-02 21:56:06.480512173 +0000 UTC m=+1313.642607294" Nov 02 21:56:06.544151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.544119 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:06.617740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.616187 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q\" (UniqueName: \"kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q\") pod \"5cc9149b-5525-402f-b190-686b259d0975\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " Nov 02 21:56:06.617740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.616701 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca\") pod \"5cc9149b-5525-402f-b190-686b259d0975\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " Nov 02 21:56:06.617740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.617149 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca\") pod \"5cc9149b-5525-402f-b190-686b259d0975\" (UID: \"5cc9149b-5525-402f-b190-686b259d0975\") " Nov 02 21:56:06.617740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.617553 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca" pod "5cc9149b-5525-402f-b190-686b259d0975" (UID: "5cc9149b-5525-402f-b190-686b259d0975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:06.621908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.618216 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc9149b-5525-402f-b190-686b259d0975-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:06.628713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.622292 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q" pod "5cc9149b-5525-402f-b190-686b259d0975" (UID: "5cc9149b-5525-402f-b190-686b259d0975"). InnerVolumeSpecName "kube-api-access-jxs7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:06.628713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.622610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca" pod "5cc9149b-5525-402f-b190-686b259d0975" (UID: "5cc9149b-5525-402f-b190-686b259d0975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:06.720416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.720376 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxs7q\" (UniqueName: \"kubernetes.io/projected/5cc9149b-5525-402f-b190-686b259d0975-kube-api-access-jxs7q\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:06.720416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:06.720417 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc9149b-5525-402f-b190-686b259d0975-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:07.115576 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:07.115525 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:56:07.378225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.378194 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" Nov 02 21:56:07.378225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.378213 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8" event={"ID":"5cc9149b-5525-402f-b190-686b259d0975","Type":"ContainerDied","Data":"fe62b0c563bc041e1f9debf7cfdc2488cbbc87787361b648e3f7b4f28e6065e8"} Nov 02 21:56:07.378817 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.378267 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe62b0c563bc041e1f9debf7cfdc2488cbbc87787361b648e3f7b4f28e6065e8" Nov 02 21:56:07.432349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.431436 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:56:07.432349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.432294 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:56:07.433293 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.433269 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 21:56:07.554206 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.554178 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:07.634110 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.634036 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca\") pod \"963a778f-929d-4353-9087-75f7e9cb1058\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " Nov 02 21:56:07.634265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.634237 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm\" (UniqueName: \"kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm\") pod \"963a778f-929d-4353-9087-75f7e9cb1058\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " Nov 02 21:56:07.634335 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.634270 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca\") pod \"963a778f-929d-4353-9087-75f7e9cb1058\" (UID: \"963a778f-929d-4353-9087-75f7e9cb1058\") " Nov 02 21:56:07.634431 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.634410 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca" pod "963a778f-929d-4353-9087-75f7e9cb1058" (UID: "963a778f-929d-4353-9087-75f7e9cb1058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:07.634888 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.634863 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:07.636415 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.636392 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca" pod "963a778f-929d-4353-9087-75f7e9cb1058" (UID: "963a778f-929d-4353-9087-75f7e9cb1058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:07.636545 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.636525 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm" pod "963a778f-929d-4353-9087-75f7e9cb1058" (UID: "963a778f-929d-4353-9087-75f7e9cb1058"). InnerVolumeSpecName "kube-api-access-scrzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:07.736375 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.736343 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scrzm\" (UniqueName: \"kubernetes.io/projected/963a778f-929d-4353-9087-75f7e9cb1058-kube-api-access-scrzm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:07.736375 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.736375 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963a778f-929d-4353-9087-75f7e9cb1058-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:07.907745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.905830 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7196-e0c15f"] Nov 02 21:56:07.913282 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:07.913251 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7196-e0c15f"] Nov 02 21:56:08.051875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.051843 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7196-ba44a2"] Nov 02 21:56:08.052537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052516 5091 cpu_manager.go:414] "containerMap: removing container" podUID="963a778f-929d-4353-9087-75f7e9cb1058" containerName="revoke-keys" Nov 02 21:56:08.052655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052538 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963a778f-929d-4353-9087-75f7e9cb1058" containerName="revoke-keys" Nov 02 21:56:08.052655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052558 5091 cpu_manager.go:414] "containerMap: removing container" podUID="571c31cd-de35-4790-9494-f9b869ff0904" containerName="test-curl-7196-e0c15f" Nov 02 21:56:08.052655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052566 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571c31cd-de35-4790-9494-f9b869ff0904" containerName="test-curl-7196-e0c15f" Nov 02 21:56:08.052655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052606 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5cc9149b-5525-402f-b190-686b259d0975" containerName="revoke-keys" Nov 02 21:56:08.052655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052614 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc9149b-5525-402f-b190-686b259d0975" containerName="revoke-keys" Nov 02 21:56:08.052905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052793 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="963a778f-929d-4353-9087-75f7e9cb1058" containerName="revoke-keys" Nov 02 21:56:08.052905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052807 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="571c31cd-de35-4790-9494-f9b869ff0904" containerName="test-curl-7196-e0c15f" Nov 02 21:56:08.052905 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.052825 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc9149b-5525-402f-b190-686b259d0975" containerName="revoke-keys" Nov 02 21:56:08.056796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.056776 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:08.111666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.111637 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7196-ba44a2"] Nov 02 21:56:08.142225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.142198 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdqx\" (UniqueName: \"kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx\") pod \"test-curl-7196-ba44a2\" (UID: \"30810da5-de42-4c99-b47d-d0d07ddfc1fd\") " pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:08.243382 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.243312 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdqx\" (UniqueName: \"kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx\") pod \"test-curl-7196-ba44a2\" (UID: \"30810da5-de42-4c99-b47d-d0d07ddfc1fd\") " pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:08.300420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.300397 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdqx\" (UniqueName: \"kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx\") pod \"test-curl-7196-ba44a2\" (UID: \"30810da5-de42-4c99-b47d-d0d07ddfc1fd\") " pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:08.349317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.349290 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc"] Nov 02 21:56:08.353813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.353797 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.370087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.370066 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:08.391266 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.391232 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7196-e0c15f" podUID="571c31cd-de35-4790-9494-f9b869ff0904" containerName="test-curl-7196-e0c15f" containerID="cri-o://765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712" gracePeriod=2 Nov 02 21:56:08.391665 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.391609 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" Nov 02 21:56:08.399798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.395843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v" event={"ID":"963a778f-929d-4353-9087-75f7e9cb1058","Type":"ContainerDied","Data":"54da9cf95b3463340df3c6b4463c62fb138455fa938bdea9d0a207a7b4abd3ae"} Nov 02 21:56:08.399798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.395895 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54da9cf95b3463340df3c6b4463c62fb138455fa938bdea9d0a207a7b4abd3ae" Nov 02 21:56:08.410408 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.410383 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc"] Nov 02 21:56:08.446334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.445751 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282c8\" (UniqueName: \"kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.446334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.445884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.446334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.445991 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.517784 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:08.517703 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-e0c15f\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="571c31cd-de35-4790-9494-f9b869ff0904" pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:56:08.546931 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.546904 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.547103 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.547026 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.547342 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.547309 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282c8\" (UniqueName: \"kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.547806 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.547785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.549573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.549541 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.606564 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.606534 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-282c8\" (UniqueName: \"kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8\") pod \"maas-api-revoke-sub-3cd9eb87-2k7kc\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.671481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.671453 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:08.759766 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:08.759737 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7196-ba44a2"] Nov 02 21:56:08.761973 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:08.761943 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30810da5_de42_4c99_b47d_d0d07ddfc1fd.slice/crio-03b19eba80468bab6bdd9439beffcec7dddc2e13ad2c2e86fbff25a5bae1857e WatchSource:0}: Error finding container 03b19eba80468bab6bdd9439beffcec7dddc2e13ad2c2e86fbff25a5bae1857e: Status 404 returned error can't find the container with id 03b19eba80468bab6bdd9439beffcec7dddc2e13ad2c2e86fbff25a5bae1857e Nov 02 21:56:09.056722 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:09.056642 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df4cf88_7749_4d55_b258_b6499342e7cf.slice/crio-a7f0e227c3bcd3ca17e57a20e1f850f0e672c6315d35fcc990660d5554eb3396 WatchSource:0}: Error finding container a7f0e227c3bcd3ca17e57a20e1f850f0e672c6315d35fcc990660d5554eb3396: Status 404 returned error can't find the container with id a7f0e227c3bcd3ca17e57a20e1f850f0e672c6315d35fcc990660d5554eb3396 Nov 02 21:56:09.058468 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.058422 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc"] Nov 02 21:56:09.064390 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.064368 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2"] Nov 02 21:56:09.068819 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.068798 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e799f06-m7mn2"] Nov 02 21:56:09.400846 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.400808 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" event={"ID":"5df4cf88-7749-4d55-b258-b6499342e7cf","Type":"ContainerStarted","Data":"b339749f1220fa6cc79d6a74b1ccb6bef338ad89a5ba745ef9807a4bc14f7504"} Nov 02 21:56:09.401224 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.400873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" event={"ID":"5df4cf88-7749-4d55-b258-b6499342e7cf","Type":"ContainerStarted","Data":"a7f0e227c3bcd3ca17e57a20e1f850f0e672c6315d35fcc990660d5554eb3396"} Nov 02 21:56:09.402612 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.402582 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7196-ba44a2" event={"ID":"30810da5-de42-4c99-b47d-d0d07ddfc1fd","Type":"ContainerStarted","Data":"ec48305b93e226bff19ce0040e11b91ba77c82797db1218e37ebb64605f25935"} Nov 02 21:56:09.402729 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.402621 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7196-ba44a2" event={"ID":"30810da5-de42-4c99-b47d-d0d07ddfc1fd","Type":"ContainerStarted","Data":"03b19eba80468bab6bdd9439beffcec7dddc2e13ad2c2e86fbff25a5bae1857e"} Nov 02 21:56:09.457196 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.457170 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq"] Nov 02 21:56:09.461701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.461683 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.517885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.517858 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq"] Nov 02 21:56:09.519932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.519881 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" podStartSLOduration=1.5198650420000002 podStartE2EDuration="1.519865042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:09.515262486 +0000 UTC m=+1316.677357607" watchObservedRunningTime="2026-11-02 21:56:09.519865042 +0000 UTC m=+1316.681960165" Nov 02 21:56:09.562155 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.562114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.562335 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.562166 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.562534 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.562502 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k4b\" (UniqueName: \"kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.632867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.632768 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7196-ba44a2" podStartSLOduration=1.632755255 podStartE2EDuration="1.632755255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:09.625913817 +0000 UTC m=+1316.788008945" watchObservedRunningTime="2026-11-02 21:56:09.632755255 +0000 UTC m=+1316.794850374" Nov 02 21:56:09.664558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.664521 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k4b\" (UniqueName: \"kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.664759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.664710 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.664759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.664745 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.666040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.666007 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.668100 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.668071 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.720849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.720828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k4b\" (UniqueName: \"kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b\") pod \"maas-api-revoke-sub-15ec0d1a-qgwqq\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:09.776321 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:09.776294 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:10.153876 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:10.153847 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf15d935_9a83_4729_9a65_461e62183464.slice/crio-851c72b31ad308e6fb2c6cfb29effc399dbd312ce2fbb456d0406ebed3ce7fec WatchSource:0}: Error finding container 851c72b31ad308e6fb2c6cfb29effc399dbd312ce2fbb456d0406ebed3ce7fec: Status 404 returned error can't find the container with id 851c72b31ad308e6fb2c6cfb29effc399dbd312ce2fbb456d0406ebed3ce7fec Nov 02 21:56:10.155747 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.155605 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq"] Nov 02 21:56:10.413001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.412975 5091 generic.go:356] "Generic (PLEG): container finished" podID="5df4cf88-7749-4d55-b258-b6499342e7cf" containerID="b339749f1220fa6cc79d6a74b1ccb6bef338ad89a5ba745ef9807a4bc14f7504" exitCode=0 Nov 02 21:56:10.413361 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.413055 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" event={"ID":"5df4cf88-7749-4d55-b258-b6499342e7cf","Type":"ContainerDied","Data":"b339749f1220fa6cc79d6a74b1ccb6bef338ad89a5ba745ef9807a4bc14f7504"} Nov 02 21:56:10.414645 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.414619 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" event={"ID":"af15d935-9a83-4729-9a65-461e62183464","Type":"ContainerStarted","Data":"851c72b31ad308e6fb2c6cfb29effc399dbd312ce2fbb456d0406ebed3ce7fec"} Nov 02 21:56:10.679233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.679202 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:56:10.780236 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.780163 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp\" (UniqueName: \"kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp\") pod \"571c31cd-de35-4790-9494-f9b869ff0904\" (UID: \"571c31cd-de35-4790-9494-f9b869ff0904\") " Nov 02 21:56:10.782255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.782225 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp" pod "571c31cd-de35-4790-9494-f9b869ff0904" (UID: "571c31cd-de35-4790-9494-f9b869ff0904"). InnerVolumeSpecName "kube-api-access-fzcqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:10.881718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.881657 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcqp\" (UniqueName: \"kubernetes.io/projected/571c31cd-de35-4790-9494-f9b869ff0904-kube-api-access-fzcqp\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:10.979159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.979128 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571c31cd-de35-4790-9494-f9b869ff0904" path="/var/lib/kubelet/pods/571c31cd-de35-4790-9494-f9b869ff0904/volumes" Nov 02 21:56:10.979695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:10.979666 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07e55ce-ad20-4045-bbc6-ebe66bb80ec3" path="/var/lib/kubelet/pods/f07e55ce-ad20-4045-bbc6-ebe66bb80ec3/volumes" Nov 02 21:56:11.107834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.107801 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7196-ba44a2"] Nov 02 21:56:11.117650 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.113037 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7196-ba44a2"] Nov 02 21:56:11.427789 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.427754 5091 generic.go:356] "Generic (PLEG): container finished" podID="af15d935-9a83-4729-9a65-461e62183464" containerID="1edf107b46d5f8379f3b5d1449828622c12f5d11e0d2d647e0539d3ba1046f6b" exitCode=0 Nov 02 21:56:11.428217 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.427836 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" event={"ID":"af15d935-9a83-4729-9a65-461e62183464","Type":"ContainerDied","Data":"1edf107b46d5f8379f3b5d1449828622c12f5d11e0d2d647e0539d3ba1046f6b"} Nov 02 21:56:11.429282 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.429259 5091 generic.go:356] "Generic (PLEG): container finished" podID="571c31cd-de35-4790-9494-f9b869ff0904" containerID="765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712" exitCode=137 Nov 02 21:56:11.429408 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.429295 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-e0c15f" Nov 02 21:56:11.429408 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.429340 5091 scope.go:122] "RemoveContainer" containerID="765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712" Nov 02 21:56:11.429556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.429532 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7196-ba44a2" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" containerName="test-curl-7196-ba44a2" containerID="cri-o://ec48305b93e226bff19ce0040e11b91ba77c82797db1218e37ebb64605f25935" gracePeriod=2 Nov 02 21:56:11.440822 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.440800 5091 scope.go:122] "RemoveContainer" containerID="765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712" Nov 02 21:56:11.441125 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:11.441102 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712\": container with ID starting with 765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712 not found: ID does not exist" containerID="765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712" Nov 02 21:56:11.441183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.441133 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712"} err="failed to get container status \"765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712\": rpc error: code = NotFound desc = could not find container \"765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712\": container with ID starting with 765218c1d1e0fb6a4e64350a557f57faff7f2906435a19730973120e6a7ef712 not found: ID does not exist" Nov 02 21:56:11.590803 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:11.590769 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:11.610053 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.610028 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:11.661148 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:11.661112 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:11.690099 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.690026 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca\") pod \"5df4cf88-7749-4d55-b258-b6499342e7cf\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " Nov 02 21:56:11.690099 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.690083 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8\" (UniqueName: \"kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8\") pod \"5df4cf88-7749-4d55-b258-b6499342e7cf\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " Nov 02 21:56:11.690338 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.690316 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca\") pod \"5df4cf88-7749-4d55-b258-b6499342e7cf\" (UID: \"5df4cf88-7749-4d55-b258-b6499342e7cf\") " Nov 02 21:56:11.690777 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.690740 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca" pod "5df4cf88-7749-4d55-b258-b6499342e7cf" (UID: "5df4cf88-7749-4d55-b258-b6499342e7cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:11.691030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.691010 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:11.692487 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.692406 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca" pod "5df4cf88-7749-4d55-b258-b6499342e7cf" (UID: "5df4cf88-7749-4d55-b258-b6499342e7cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:11.692580 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.692476 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8" pod "5df4cf88-7749-4d55-b258-b6499342e7cf" (UID: "5df4cf88-7749-4d55-b258-b6499342e7cf"). InnerVolumeSpecName "kube-api-access-282c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:11.792416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.792378 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df4cf88-7749-4d55-b258-b6499342e7cf-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:11.792416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:11.792412 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282c8\" (UniqueName: \"kubernetes.io/projected/5df4cf88-7749-4d55-b258-b6499342e7cf-kube-api-access-282c8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:12.450695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.450549 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5"] Nov 02 21:56:12.456271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.454952 5091 cpu_manager.go:414] "containerMap: removing container" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" containerName="test-curl-7196-ba44a2" Nov 02 21:56:12.456271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.455012 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" containerName="test-curl-7196-ba44a2" Nov 02 21:56:12.456271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.455058 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5df4cf88-7749-4d55-b258-b6499342e7cf" containerName="revoke-keys" Nov 02 21:56:12.456271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.455067 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df4cf88-7749-4d55-b258-b6499342e7cf" containerName="revoke-keys" Nov 02 21:56:12.456271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.455310 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df4cf88-7749-4d55-b258-b6499342e7cf" containerName="revoke-keys" Nov 02 21:56:12.456271 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.455341 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" containerName="test-curl-7196-ba44a2" Nov 02 21:56:12.477076 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.477036 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.528307 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:12.528232 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:12.558688 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.554841 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5"] Nov 02 21:56:12.588187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.588157 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" Nov 02 21:56:12.589395 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.589328 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n"] Nov 02 21:56:12.609563 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.609535 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.609965 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.609943 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.610182 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.610168 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzhj\" (UniqueName: \"kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.618425 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.618396 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc" event={"ID":"5df4cf88-7749-4d55-b258-b6499342e7cf","Type":"ContainerDied","Data":"a7f0e227c3bcd3ca17e57a20e1f850f0e672c6315d35fcc990660d5554eb3396"} Nov 02 21:56:12.618694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.618663 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7f0e227c3bcd3ca17e57a20e1f850f0e672c6315d35fcc990660d5554eb3396" Nov 02 21:56:12.618811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.618633 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.631658 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.631626 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n"] Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:12.705185 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.712724 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn6fb\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.712803 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.712857 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.712901 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.712964 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzhj\" (UniqueName: \"kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713080 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713111 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713139 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713180 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713223 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.713320 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.714962 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.720160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.719368 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.805611 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.805542 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzhj\" (UniqueName: \"kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj\") pod \"maas-api-revoke-sub-008face6-sjws5\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.814867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.814828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6fb\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.814912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.814964 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815003 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815157 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815192 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815232 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815337 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.815505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.816132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.816818 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.817727 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.817077 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.818475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.818360 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.818714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.818608 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.821746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.821700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:12.841304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.836103 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv"] Nov 02 21:56:12.845476 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.843873 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:12.867239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.866308 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:12.996416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.941690 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:12.996416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.941807 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:12.996416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.941866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:12.996416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.941936 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:12.996416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:12.991252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.009631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.009332 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.009631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.009372 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.009631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.009410 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4h6\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.009631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.009485 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.027702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.022958 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6fb\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:13.097899 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.097695 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:13.101723 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:13.101297 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:13.110831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.110797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.110981 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.110870 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.110981 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.110937 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.111087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.111087 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111080 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.111193 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.111193 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111175 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4h6\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.111301 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111276 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.111353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111333 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.112089 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.111785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.113239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.112414 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.113239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.112488 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv"] Nov 02 21:56:13.113239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.112585 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:13.113239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.112974 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.113239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.113167 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.113239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.113234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.114844 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.114814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.115334 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.115308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.162542 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:13.162379 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:13.170596 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.170548 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.171146 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.171125 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4h6\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.209056 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.209019 5091 scope.go:122] "RemoveContainer" containerID="804061ab55f3b31cd1271096f2d35f18a00b48b0bd59052b7a30a42b11363385" Nov 02 21:56:13.212179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.212147 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b\" (UniqueName: \"kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b\") pod \"af15d935-9a83-4729-9a65-461e62183464\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " Nov 02 21:56:13.212400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.212381 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca\") pod \"af15d935-9a83-4729-9a65-461e62183464\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " Nov 02 21:56:13.212469 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.212430 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca\") pod \"af15d935-9a83-4729-9a65-461e62183464\" (UID: \"af15d935-9a83-4729-9a65-461e62183464\") " Nov 02 21:56:13.213184 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.213157 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca" pod "af15d935-9a83-4729-9a65-461e62183464" (UID: "af15d935-9a83-4729-9a65-461e62183464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:13.217921 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.217863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca" pod "af15d935-9a83-4729-9a65-461e62183464" (UID: "af15d935-9a83-4729-9a65-461e62183464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:13.217921 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.217897 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b" pod "af15d935-9a83-4729-9a65-461e62183464" (UID: "af15d935-9a83-4729-9a65-461e62183464"). InnerVolumeSpecName "kube-api-access-p4k4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:13.224657 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.224629 5091 scope.go:122] "RemoveContainer" containerID="a0b3a648d525932e439d900358633af3636f1d3a73258db84640fd8d5fe5dc5a" Nov 02 21:56:13.241709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.241659 5091 scope.go:122] "RemoveContainer" containerID="df73fd6ccada05ca6b1ec717a79793c0b783bc33f76ec0e3771f504363f721b9" Nov 02 21:56:13.247294 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.247047 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:13.265421 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.265386 5091 scope.go:122] "RemoveContainer" containerID="6db9b86202db07b7871c16c503e9f6230fafc57ce554c9e441f54364d29001f6" Nov 02 21:56:13.316740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.316647 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15d935-9a83-4729-9a65-461e62183464-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:13.316740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.316697 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15d935-9a83-4729-9a65-461e62183464-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:13.316740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.316715 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k4b\" (UniqueName: \"kubernetes.io/projected/af15d935-9a83-4729-9a65-461e62183464-kube-api-access-p4k4b\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:13.469141 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.469105 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:13.481566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.481521 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5"] Nov 02 21:56:13.489172 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:13.489136 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda252fa78_d4b5_4852_bfff_41fa96f8699b.slice/crio-e444d51fca890c396c18f376cf23e023d04b40262e4f80eaa5318927e0a48911 WatchSource:0}: Error finding container e444d51fca890c396c18f376cf23e023d04b40262e4f80eaa5318927e0a48911: Status 404 returned error can't find the container with id e444d51fca890c396c18f376cf23e023d04b40262e4f80eaa5318927e0a48911 Nov 02 21:56:13.608830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.608802 5091 generic.go:356] "Generic (PLEG): container finished" podID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" containerID="ec48305b93e226bff19ce0040e11b91ba77c82797db1218e37ebb64605f25935" exitCode=137 Nov 02 21:56:13.613931 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.613817 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" event={"ID":"a252fa78-d4b5-4852-bfff-41fa96f8699b","Type":"ContainerStarted","Data":"e444d51fca890c396c18f376cf23e023d04b40262e4f80eaa5318927e0a48911"} Nov 02 21:56:13.622385 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.616098 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" event={"ID":"af15d935-9a83-4729-9a65-461e62183464","Type":"ContainerDied","Data":"851c72b31ad308e6fb2c6cfb29effc399dbd312ce2fbb456d0406ebed3ce7fec"} Nov 02 21:56:13.622385 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.616146 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="851c72b31ad308e6fb2c6cfb29effc399dbd312ce2fbb456d0406ebed3ce7fec" Nov 02 21:56:13.622385 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.616197 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq" Nov 02 21:56:13.686453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.686295 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n"] Nov 02 21:56:13.699112 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.699058 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.699224 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.699207 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.699300 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.699288 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.721970 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.721944 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:13.825152 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.825125 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx\" (UniqueName: \"kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx\") pod \"30810da5-de42-4c99-b47d-d0d07ddfc1fd\" (UID: \"30810da5-de42-4c99-b47d-d0d07ddfc1fd\") " Nov 02 21:56:13.827330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.827303 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx" pod "30810da5-de42-4c99-b47d-d0d07ddfc1fd" (UID: "30810da5-de42-4c99-b47d-d0d07ddfc1fd"). InnerVolumeSpecName "kube-api-access-6rdqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:13.878712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.878074 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.878712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.878149 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.878712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.878177 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.886761 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.884216 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv"] Nov 02 21:56:13.927613 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:13.927585 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rdqx\" (UniqueName: \"kubernetes.io/projected/30810da5-de42-4c99-b47d-d0d07ddfc1fd-kube-api-access-6rdqx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:14.628135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.628095 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" event={"ID":"592d0e95-5afb-4f73-8c3e-6594e08388c2","Type":"ContainerStarted","Data":"7d89cc3448e6ce41cecbf1df20cdfeb09f5006fcda1f0937699d8011c9e398cc"} Nov 02 21:56:14.628603 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.628163 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" event={"ID":"592d0e95-5afb-4f73-8c3e-6594e08388c2","Type":"ContainerStarted","Data":"7a2444a404d08fb108e92bd4f881bce519651d1f003d5eaa3859ec5fc401bb6c"} Nov 02 21:56:14.630179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.630144 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" event={"ID":"0709d9c9-5704-435e-909a-ad40dfdb78e0","Type":"ContainerStarted","Data":"4bc0eb5b8d0b96e9d9548e876191d59be919489c58057330c6394a08077d3dea"} Nov 02 21:56:14.630309 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.630203 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" event={"ID":"0709d9c9-5704-435e-909a-ad40dfdb78e0","Type":"ContainerStarted","Data":"bfb5ddfe817ed4ff9063d2387e5e147c816cfa19d3abe02654ba9a1d54c7896d"} Nov 02 21:56:14.632099 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.632075 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:14.632099 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.632090 5091 scope.go:122] "RemoveContainer" containerID="ec48305b93e226bff19ce0040e11b91ba77c82797db1218e37ebb64605f25935" Nov 02 21:56:14.634415 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.634377 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" event={"ID":"a252fa78-d4b5-4852-bfff-41fa96f8699b","Type":"ContainerStarted","Data":"6226f6792bdd5c677fa0694b82941687cb0c9705b3f8240b9982fb286a836e9a"} Nov 02 21:56:14.750286 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.750226 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podStartSLOduration=2.750206416 podStartE2EDuration="2.750206416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:14.74229292 +0000 UTC m=+1321.904388037" watchObservedRunningTime="2026-11-02 21:56:14.750206416 +0000 UTC m=+1321.912301536" Nov 02 21:56:14.793461 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:14.793417 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-ba44a2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" pod="opendatahub/test-curl-7196-ba44a2" Nov 02 21:56:14.926419 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.926284 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" podStartSLOduration=2.926259343 podStartE2EDuration="2.926259343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:14.909187363 +0000 UTC m=+1322.071282522" watchObservedRunningTime="2026-11-02 21:56:14.926259343 +0000 UTC m=+1322.088354465" Nov 02 21:56:14.999693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:14.998969 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30810da5-de42-4c99-b47d-d0d07ddfc1fd" path="/var/lib/kubelet/pods/30810da5-de42-4c99-b47d-d0d07ddfc1fd/volumes" Nov 02 21:56:15.029394 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.029154 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podStartSLOduration=3.029139304 podStartE2EDuration="3.029139304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:15.027661244 +0000 UTC m=+1322.189756573" watchObservedRunningTime="2026-11-02 21:56:15.029139304 +0000 UTC m=+1322.191234425" Nov 02 21:56:15.248211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.248122 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:15.249778 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.249751 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:15.249879 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.249803 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:15.473082 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.473041 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:15.480695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.477790 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:15.480695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.477850 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:15.650214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.650185 5091 generic.go:356] "Generic (PLEG): container finished" podID="a252fa78-d4b5-4852-bfff-41fa96f8699b" containerID="6226f6792bdd5c677fa0694b82941687cb0c9705b3f8240b9982fb286a836e9a" exitCode=0 Nov 02 21:56:15.650633 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:15.650276 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" event={"ID":"a252fa78-d4b5-4852-bfff-41fa96f8699b","Type":"ContainerDied","Data":"6226f6792bdd5c677fa0694b82941687cb0c9705b3f8240b9982fb286a836e9a"} Nov 02 21:56:16.259698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:16.256198 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:16.259698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:16.256288 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:16.478693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:16.473131 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:16.478693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:16.473194 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:17.342504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.332463 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:17.342504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.333377 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:17.437745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.437716 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:17.495693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.487801 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:17.495693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.487865 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:17.532857 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.532808 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca\") pod \"a252fa78-d4b5-4852-bfff-41fa96f8699b\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " Nov 02 21:56:17.533050 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.533038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj\" (UniqueName: \"kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj\") pod \"a252fa78-d4b5-4852-bfff-41fa96f8699b\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " Nov 02 21:56:17.533153 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.533133 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca\") pod \"a252fa78-d4b5-4852-bfff-41fa96f8699b\" (UID: \"a252fa78-d4b5-4852-bfff-41fa96f8699b\") " Nov 02 21:56:17.534104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.534076 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca" pod "a252fa78-d4b5-4852-bfff-41fa96f8699b" (UID: "a252fa78-d4b5-4852-bfff-41fa96f8699b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:17.536234 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.536206 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca" pod "a252fa78-d4b5-4852-bfff-41fa96f8699b" (UID: "a252fa78-d4b5-4852-bfff-41fa96f8699b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:17.537417 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.537390 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj" pod "a252fa78-d4b5-4852-bfff-41fa96f8699b" (UID: "a252fa78-d4b5-4852-bfff-41fa96f8699b"). InnerVolumeSpecName "kube-api-access-fzzhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:17.621439 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.621402 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr"] Nov 02 21:56:17.622162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.622139 5091 cpu_manager.go:414] "containerMap: removing container" podUID="af15d935-9a83-4729-9a65-461e62183464" containerName="revoke-keys" Nov 02 21:56:17.622162 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.622161 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af15d935-9a83-4729-9a65-461e62183464" containerName="revoke-keys" Nov 02 21:56:17.622341 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.622191 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a252fa78-d4b5-4852-bfff-41fa96f8699b" containerName="revoke-keys" Nov 02 21:56:17.622341 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.622198 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a252fa78-d4b5-4852-bfff-41fa96f8699b" containerName="revoke-keys" Nov 02 21:56:17.622440 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.622396 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="af15d935-9a83-4729-9a65-461e62183464" containerName="revoke-keys" Nov 02 21:56:17.622440 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.622415 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a252fa78-d4b5-4852-bfff-41fa96f8699b" containerName="revoke-keys" Nov 02 21:56:17.626388 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.626367 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.635872 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.635952 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96d2w\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636037 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636102 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636257 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636312 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.636371 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.638895 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzzhj\" (UniqueName: \"kubernetes.io/projected/a252fa78-d4b5-4852-bfff-41fa96f8699b-kube-api-access-fzzhj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.638923 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:17.639256 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.638941 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252fa78-d4b5-4852-bfff-41fa96f8699b-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:17.723699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.717047 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fb2343fe-openshift-default-dockercfg-cz9th\"" Nov 02 21:56:17.738742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.738709 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr"] Nov 02 21:56:17.739858 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.739798 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740066 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740046 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96d2w\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740327 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740313 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740482 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740734 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740717 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740853 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.740982 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.740966 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.741133 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.741115 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.741785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.741763 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.742459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.742437 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.742826 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.742808 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.743632 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.743611 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.744024 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.744003 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.746980 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.746954 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.747316 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.747301 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.790330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.785011 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" event={"ID":"a252fa78-d4b5-4852-bfff-41fa96f8699b","Type":"ContainerDied","Data":"e444d51fca890c396c18f376cf23e023d04b40262e4f80eaa5318927e0a48911"} Nov 02 21:56:17.790330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.785078 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e444d51fca890c396c18f376cf23e023d04b40262e4f80eaa5318927e0a48911" Nov 02 21:56:17.790330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.785150 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5" Nov 02 21:56:17.823709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.823658 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96d2w\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.829280 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.829251 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token\") pod \"e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:17.972700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:17.972348 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:18.065894 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.062418 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr"] Nov 02 21:56:18.173177 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.172213 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z"] Nov 02 21:56:18.203149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.203078 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d3c53f-z8wrr"] Nov 02 21:56:18.250742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.243044 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39527c9-zhj2z"] Nov 02 21:56:18.257691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.255780 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:18.257691 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.255840 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:18.472091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.471621 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:18.472091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.471697 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:18.740131 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:18.740085 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544748cb_be52_4ce3_8772_78998f8b5fe8.slice/crio-5f91898f8adc40b0a1651fcd502b1c249631a14f6af995abf8671964fa9f2726 WatchSource:0}: Error finding container 5f91898f8adc40b0a1651fcd502b1c249631a14f6af995abf8671964fa9f2726: Status 404 returned error can't find the container with id 5f91898f8adc40b0a1651fcd502b1c249631a14f6af995abf8671964fa9f2726 Nov 02 21:56:18.743463 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.743440 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr"] Nov 02 21:56:18.743622 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.743481 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:18.743779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.743756 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:18.743889 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.743876 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:18.858329 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:18.858297 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" event={"ID":"544748cb-be52-4ce3-8772-78998f8b5fe8","Type":"ContainerStarted","Data":"5f91898f8adc40b0a1651fcd502b1c249631a14f6af995abf8671964fa9f2726"} Nov 02 21:56:19.051422 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:19.051024 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74127a37-3725-40a7-99cd-73da4c359a28" path="/var/lib/kubelet/pods/74127a37-3725-40a7-99cd-73da4c359a28/volumes" Nov 02 21:56:19.051831 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:19.051763 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b0cb4c-f277-46e6-8e82-22b1ccb6c006" path="/var/lib/kubelet/pods/f6b0cb4c-f277-46e6-8e82-22b1ccb6c006/volumes" Nov 02 21:56:19.247775 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:19.247737 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:19.248036 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:19.248001 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:19.482833 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:19.482037 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:19.482833 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:19.482101 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:20.063372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.063316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" event={"ID":"544748cb-be52-4ce3-8772-78998f8b5fe8","Type":"ContainerStarted","Data":"b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712"} Nov 02 21:56:20.325793 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.315242 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:20.326186 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.326115 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:20.500705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.499962 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:20.500705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.500033 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:20.966557 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.966526 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:20.998620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:20.998590 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:21.086020 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.085913 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:21.087359 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.087327 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:56:21.117330 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.117264 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podStartSLOduration=4.117243167 podStartE2EDuration="4.117243167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:20.285858413 +0000 UTC m=+1327.447953534" watchObservedRunningTime="2026-11-02 21:56:21.117243167 +0000 UTC m=+1328.279338288" Nov 02 21:56:21.248873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.248365 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 21:56:21.248873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.248422 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 21:56:21.469966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.469928 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 02 21:56:21.470185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:21.470155 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 02 21:56:22.442031 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:22.441835 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:22.442825 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:22.442799 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:22.443510 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:22.443489 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 21:56:22.539924 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:22.539892 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:22.540274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:22.540249 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:22.541250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:22.541227 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 21:56:23.335600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.335565 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t"] Nov 02 21:56:23.342818 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.342792 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.388302 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.388269 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t"] Nov 02 21:56:23.394785 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.394757 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:56:23.394969 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.394773 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:56:23.395054 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.394840 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:56:23.395054 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.394862 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:56:23.431844 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.431812 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.432019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.431957 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbx9\" (UniqueName: \"kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.432091 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.432076 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.533018 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.532981 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.533474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.533121 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbx9\" (UniqueName: \"kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.533474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.533272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.533772 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.533748 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.536070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.536045 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.589090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.589033 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbx9\" (UniqueName: \"kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9\") pod \"maas-api-revoke-sub-04cff5fd-lbh2t\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:23.659655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:23.659622 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:24.036487 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:24.036453 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ec18ee_a8f0_429b_b39c_1e4854607b5f.slice/crio-5d6e68519301e7d8991256da30ee5b6e3631eb5c88ad87a50efd575b676e74d7 WatchSource:0}: Error finding container 5d6e68519301e7d8991256da30ee5b6e3631eb5c88ad87a50efd575b676e74d7: Status 404 returned error can't find the container with id 5d6e68519301e7d8991256da30ee5b6e3631eb5c88ad87a50efd575b676e74d7 Nov 02 21:56:24.039868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:24.039844 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t"] Nov 02 21:56:24.130524 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:24.130486 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" event={"ID":"68ec18ee-a8f0-429b-b39c-1e4854607b5f","Type":"ContainerStarted","Data":"5d6e68519301e7d8991256da30ee5b6e3631eb5c88ad87a50efd575b676e74d7"} Nov 02 21:56:24.288829 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:24.288742 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx"] Nov 02 21:56:24.288988 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:24.288963 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" containerID="cri-o://b5b1a25ae083faab5d21d54d43db2e5514006f43e508d2ccd0527c8aabfd2e61" gracePeriod=30 Nov 02 21:56:25.059094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:25.059062 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r"] Nov 02 21:56:25.067129 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:25.067094 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6fc593-pw87r"] Nov 02 21:56:25.145148 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:25.145110 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" event={"ID":"68ec18ee-a8f0-429b-b39c-1e4854607b5f","Type":"ContainerStarted","Data":"111be525fe6fe2ea5d01fa0e2af89f58dd66eb6f85ef831ebfb845e7f11c378c"} Nov 02 21:56:25.262694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:25.262440 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" podStartSLOduration=2.262421442 podStartE2EDuration="2.262421442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:25.25735378 +0000 UTC m=+1332.419448900" watchObservedRunningTime="2026-11-02 21:56:25.262421442 +0000 UTC m=+1332.424516561" Nov 02 21:56:26.158088 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:26.158051 5091 generic.go:356] "Generic (PLEG): container finished" podID="68ec18ee-a8f0-429b-b39c-1e4854607b5f" containerID="111be525fe6fe2ea5d01fa0e2af89f58dd66eb6f85ef831ebfb845e7f11c378c" exitCode=0 Nov 02 21:56:26.158496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:26.158144 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" event={"ID":"68ec18ee-a8f0-429b-b39c-1e4854607b5f","Type":"ContainerDied","Data":"111be525fe6fe2ea5d01fa0e2af89f58dd66eb6f85ef831ebfb845e7f11c378c"} Nov 02 21:56:26.987401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:26.987365 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d255fd8-46f0-485c-b5a2-19d304bd4e6b" path="/var/lib/kubelet/pods/8d255fd8-46f0-485c-b5a2-19d304bd4e6b/volumes" Nov 02 21:56:27.557480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:27.543114 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5wrqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:56:27.557480 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:27.543174 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:56:28.363692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.363219 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h"] Nov 02 21:56:28.772692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.768752 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.803703 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.796254 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:28.818288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.818244 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:28.880700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.866225 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtkj\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.907068 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.884582 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h"] Nov 02 21:56:28.907351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.907328 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.907494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.907478 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.907605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.907591 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.907848 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.907833 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.907975 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.907962 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.908259 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.908244 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.908400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.908387 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.908527 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.908511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:28.928179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:28.910961 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:29.028695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.025594 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca\") pod \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " Nov 02 21:56:29.054696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.047324 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca" pod "68ec18ee-a8f0-429b-b39c-1e4854607b5f" (UID: "68ec18ee-a8f0-429b-b39c-1e4854607b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:29.058849 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.058819 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca\") pod \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " Nov 02 21:56:29.059086 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.059067 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9\" (UniqueName: \"kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9\") pod \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\" (UID: \"68ec18ee-a8f0-429b-b39c-1e4854607b5f\") " Nov 02 21:56:29.059509 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.059482 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.059715 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.059694 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.059890 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.059869 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.060080 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.060061 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtkj\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.060252 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.060235 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.060381 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.060364 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.060504 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.060489 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.060716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.060697 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.060860 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.060844 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.061160 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.061142 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:29.121164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.117576 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.121164 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.119245 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.144718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.122393 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.144718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.122554 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w"] Nov 02 21:56:29.144718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.122587 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83107559-4zs9w"] Nov 02 21:56:29.144718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.122851 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.144718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.123497 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.144718 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.125024 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca" pod "68ec18ee-a8f0-429b-b39c-1e4854607b5f" (UID: "68ec18ee-a8f0-429b-b39c-1e4854607b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:56:29.185771 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.170078 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ec18ee-a8f0-429b-b39c-1e4854607b5f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:29.189587 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.189556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtkj\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.199690 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.199376 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.200700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.199933 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.200700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.200175 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9" pod "68ec18ee-a8f0-429b-b39c-1e4854607b5f" (UID: "68ec18ee-a8f0-429b-b39c-1e4854607b5f"). InnerVolumeSpecName "kube-api-access-ftbx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:29.279706 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.277975 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftbx9\" (UniqueName: \"kubernetes.io/projected/68ec18ee-a8f0-429b-b39c-1e4854607b5f-kube-api-access-ftbx9\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:29.401074 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.384982 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.430239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.423780 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.430239 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.424286 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.488813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.480299 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.520865 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.520797 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.526995 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.525354 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.542897 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.542850 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.600997 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.600952 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:29.831746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.823768 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 21:56:29.831746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:29.824155 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:56:30.177537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.169779 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.236696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.211621 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" event={"ID":"68ec18ee-a8f0-429b-b39c-1e4854607b5f","Type":"ContainerDied","Data":"5d6e68519301e7d8991256da30ee5b6e3631eb5c88ad87a50efd575b676e74d7"} Nov 02 21:56:30.274696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.229110 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t" Nov 02 21:56:30.275047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.275018 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6e68519301e7d8991256da30ee5b6e3631eb5c88ad87a50efd575b676e74d7" Nov 02 21:56:30.452749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.448920 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.452749 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.449616 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:56:30.469844 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.469805 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.479158 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.479133 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:56:30.536711 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.526172 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 02 21:56:30.536711 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.527022 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:56:30.539778 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.539737 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.777005 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.776808 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.926701 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.924786 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:56:30.941663 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:30.941621 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 21:56:31.133747 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.132100 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:56:31.175971 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.175923 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 21:56:31.234077 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.234029 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.293562 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.293509 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.323244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.323206 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c86be35-26f4-42dc-b485-e161cf95abba" path="/var/lib/kubelet/pods/9c86be35-26f4-42dc-b485-e161cf95abba/volumes" Nov 02 21:56:31.324036 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.324014 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:56:31.324279 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.324263 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:56:31.329754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.329736 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:56:31.353213 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.353187 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd"] Nov 02 21:56:31.354520 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.354494 5091 cpu_manager.go:414] "containerMap: removing container" podUID="68ec18ee-a8f0-429b-b39c-1e4854607b5f" containerName="revoke-keys" Nov 02 21:56:31.354660 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.354645 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ec18ee-a8f0-429b-b39c-1e4854607b5f" containerName="revoke-keys" Nov 02 21:56:31.354999 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.354981 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ec18ee-a8f0-429b-b39c-1e4854607b5f" containerName="revoke-keys" Nov 02 21:56:31.369152 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.369125 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd"] Nov 02 21:56:31.369472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.369432 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:31.458217 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.458183 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a16a02cb-dockercfg-fh6n5\"" Nov 02 21:56:31.458521 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.458504 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a16a02cb\"" Nov 02 21:56:31.818803 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.801126 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:56:31.880233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.880152 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.883811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.881739 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 21:56:31.883811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.882122 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:56:31.922065 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.922012 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm6w\" (UniqueName: \"kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w\") pod \"payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:31.922496 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:31.922465 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:32.063695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.059686 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm6w\" (UniqueName: \"kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w\") pod \"payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:32.063695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.059821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:32.063695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.060996 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:32.388746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.236779 5091 generic.go:356] "Generic (PLEG): container finished" podID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerID="b5b1a25ae083faab5d21d54d43db2e5514006f43e508d2ccd0527c8aabfd2e61" exitCode=0 Nov 02 21:56:32.388746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.251096 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" event={"ID":"6d69c63c-9787-40b3-a692-6f723a96cefa","Type":"ContainerDied","Data":"b5b1a25ae083faab5d21d54d43db2e5514006f43e508d2ccd0527c8aabfd2e61"} Nov 02 21:56:32.388746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.316953 5091 scope.go:122] "RemoveContainer" containerID="ec78dd3a1e8ccad850d45e5ddab65490c1547cbe9d3d9d0d8dfec3ede1e2abb4" Nov 02 21:56:32.388746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.284168 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm6w\" (UniqueName: \"kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w\") pod \"payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:32.388746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.319799 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:56:32.388746 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.372246 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.462119 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.462019 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.502218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.502171 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:32.627454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.627053 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:32.627454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.627224 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:32.627454 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:32.627334 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:33.040700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.033819 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.084810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.084766 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.085383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.085363 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt"] Nov 02 21:56:33.113620 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.113587 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:56:33.115490 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.115466 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.184198 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.180595 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:56:33.221181 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.214787 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.300715 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.275659 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:56:33.343801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.322267 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:56:33.343801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.322479 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:56:33.378260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.373752 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.412742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.412312 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.412984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.412949 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.413176 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.413159 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.413413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.413392 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.413557 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.413540 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.413742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.413724 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.413973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.413956 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.414100 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.414085 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.414697 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.414625 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmn2z\" (UniqueName: \"kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z\") pod \"payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.414869 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.414849 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.419742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.419491 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:33.420190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.419885 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:33.422662 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.422463 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "kube-api-access-7hfw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:33.423108 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.422852 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:33.428459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.428177 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:33.438050 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.437837 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:56:33.438505 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.438296 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:33.438743 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.438629 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:33.528578 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.528552 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs\") pod \"6d69c63c-9787-40b3-a692-6f723a96cefa\" (UID: \"6d69c63c-9787-40b3-a692-6f723a96cefa\") " Nov 02 21:56:33.528966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.528944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmn2z\" (UniqueName: \"kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z\") pod \"payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.529060 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.528989 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.529289 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529264 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529289 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529284 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hfw5\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-kube-api-access-7hfw5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529299 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529315 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529330 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529344 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d69c63c-9787-40b3-a692-6f723a96cefa-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529358 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.529418 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529373 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.530023 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.529996 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt"] Nov 02 21:56:33.530104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.530066 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h"] Nov 02 21:56:33.530165 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.530135 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:56:33.530217 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.530180 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:56:33.530217 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.530211 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:56:33.531503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.531392 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs" pod "6d69c63c-9787-40b3-a692-6f723a96cefa" (UID: "6d69c63c-9787-40b3-a692-6f723a96cefa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:33.534116 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.534021 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" event={"ID":"515c9af2-cd82-407e-947c-10cf076651d4","Type":"ContainerStarted","Data":"c297dc249d889d26cba3307e0ca079e8c8e9c200d6d132492fb2f94daf33545e"} Nov 02 21:56:33.535748 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.535280 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.537337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.536949 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" Nov 02 21:56:33.537337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.536995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx" event={"ID":"6d69c63c-9787-40b3-a692-6f723a96cefa","Type":"ContainerDied","Data":"fea88ce646abbfd9a9b2143b42a6e59f1572469494c18362f26623b359fcc845"} Nov 02 21:56:33.537337 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.537050 5091 scope.go:122] "RemoveContainer" containerID="b5b1a25ae083faab5d21d54d43db2e5514006f43e508d2ccd0527c8aabfd2e61" Nov 02 21:56:33.590100 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.590069 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmn2z\" (UniqueName: \"kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z\") pod \"payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.631359 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.631326 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d69c63c-9787-40b3-a692-6f723a96cefa-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:33.768982 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.768948 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:33.844012 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:33.841235 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd"] Nov 02 21:56:34.169769 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:34.169735 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a3809c_9391_4531_9285_c9233b10d30a.slice/crio-fa2266be391119bb68672df7f900d86929a39fd80e0c0fcaf13c0c23a3795c62 WatchSource:0}: Error finding container fa2266be391119bb68672df7f900d86929a39fd80e0c0fcaf13c0c23a3795c62: Status 404 returned error can't find the container with id fa2266be391119bb68672df7f900d86929a39fd80e0c0fcaf13c0c23a3795c62 Nov 02 21:56:34.170884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.170860 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt"] Nov 02 21:56:34.188840 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.188813 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx"] Nov 02 21:56:34.195377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.195354 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5wrqx"] Nov 02 21:56:34.553958 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.553871 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" event={"ID":"fb5e11da-1a68-434b-a084-709556e587f9","Type":"ContainerStarted","Data":"c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e"} Nov 02 21:56:34.554094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.553964 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" event={"ID":"fb5e11da-1a68-434b-a084-709556e587f9","Type":"ContainerStarted","Data":"1bca548fc803714bd5f628c7962bbfcc26f5d3955ad18b3d277098f178aa6fc8"} Nov 02 21:56:34.554132 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.554106 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:34.555621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.555590 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" event={"ID":"70a3809c-9391-4531-9285-c9233b10d30a","Type":"ContainerStarted","Data":"ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35"} Nov 02 21:56:34.555790 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.555643 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" event={"ID":"70a3809c-9391-4531-9285-c9233b10d30a","Type":"ContainerStarted","Data":"fa2266be391119bb68672df7f900d86929a39fd80e0c0fcaf13c0c23a3795c62"} Nov 02 21:56:34.555790 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.555707 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:34.558234 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.558214 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" event={"ID":"515c9af2-cd82-407e-947c-10cf076651d4","Type":"ContainerStarted","Data":"09010a4935ae2a047ae799041898b1d64f20d6f169af077142d32d77a5a4fc84"} Nov 02 21:56:34.605438 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.605400 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:34.606801 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.606771 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:34.607237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.606826 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:34.689782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.682194 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" podStartSLOduration=3.682172568 podStartE2EDuration="3.682172568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:34.669389336 +0000 UTC m=+1341.831484458" watchObservedRunningTime="2026-11-02 21:56:34.682172568 +0000 UTC m=+1341.844267690" Nov 02 21:56:34.794559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.794494 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" podStartSLOduration=3.794474408 podStartE2EDuration="3.794474408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:34.786153069 +0000 UTC m=+1341.948248190" watchObservedRunningTime="2026-11-02 21:56:34.794474408 +0000 UTC m=+1341.956569530" Nov 02 21:56:34.906083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:34.906032 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podStartSLOduration=7.906016627 podStartE2EDuration="7.906016627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:34.902644202 +0000 UTC m=+1342.064739324" watchObservedRunningTime="2026-11-02 21:56:34.906016627 +0000 UTC m=+1342.068111747" Nov 02 21:56:35.007698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:35.007627 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" path="/var/lib/kubelet/pods/6d69c63c-9787-40b3-a692-6f723a96cefa/volumes" Nov 02 21:56:35.623359 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:35.614133 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:35.623359 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:35.614192 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:36.605740 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:36.605708 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:36.605913 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:36.605766 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:37.119867 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:56:37.119829 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:56:37.605799 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:37.605714 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:37.605799 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:37.605765 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:38.606085 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:38.606046 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:38.606467 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:38.606114 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:39.094477 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.094440 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc"] Nov 02 21:56:39.095199 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.095182 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" Nov 02 21:56:39.095248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.095200 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" Nov 02 21:56:39.095248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.095208 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" Nov 02 21:56:39.095248 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.095214 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" Nov 02 21:56:39.095358 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.095348 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" Nov 02 21:56:39.095393 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.095360 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d69c63c-9787-40b3-a692-6f723a96cefa" containerName="istio-proxy" Nov 02 21:56:39.102810 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.102790 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.156505 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.156479 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc"] Nov 02 21:56:39.197613 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.197586 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fb2343fe-dockercfg-lhkz5\"" Nov 02 21:56:39.197898 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.197884 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fb2343fe\"" Nov 02 21:56:39.313238 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.313080 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6\") pod \"payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.313559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.313534 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.323432 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.316121 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl"] Nov 02 21:56:39.328360 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.328336 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.341927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.341898 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl"] Nov 02 21:56:39.415796 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.415756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kft\" (UniqueName: \"kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft\") pod \"payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.416049 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.415849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6\") pod \"payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.416049 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.415943 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.416179 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.416087 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume\") pod \"payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.417495 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.417467 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.501532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.501485 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6\") pod \"payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:39.519030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.518797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kft\" (UniqueName: \"kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft\") pod \"payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.519209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.519182 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume\") pod \"payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.519904 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.519878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume\") pod \"payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.606700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.606020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kft\" (UniqueName: \"kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft\") pod \"payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.606700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.606127 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:39.606700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.606173 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:39.648579 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.648542 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:39.717055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:39.717027 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:40.062734 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:40.062662 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186ef918_6b5a_4d10_b1e1_26e446e939b5.slice/crio-1bc3a3615102a7aabecd095c09f487e13c5b7af1283c5accb7eba7257498dea0 WatchSource:0}: Error finding container 1bc3a3615102a7aabecd095c09f487e13c5b7af1283c5accb7eba7257498dea0: Status 404 returned error can't find the container with id 1bc3a3615102a7aabecd095c09f487e13c5b7af1283c5accb7eba7257498dea0 Nov 02 21:56:40.063300 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.063262 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl"] Nov 02 21:56:40.162411 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:40.162379 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2736a738_51d9_4df9_bc6e_2b0257f98eb9.slice/crio-6e2a68318cd1dd9c45b8789326cc46241446fdc56b81498b66bef2087d495752 WatchSource:0}: Error finding container 6e2a68318cd1dd9c45b8789326cc46241446fdc56b81498b66bef2087d495752: Status 404 returned error can't find the container with id 6e2a68318cd1dd9c45b8789326cc46241446fdc56b81498b66bef2087d495752 Nov 02 21:56:40.165035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.165008 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc"] Nov 02 21:56:40.605107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.605072 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 02 21:56:40.605274 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.605124 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 02 21:56:40.687303 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.687266 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" event={"ID":"186ef918-6b5a-4d10-b1e1-26e446e939b5","Type":"ContainerStarted","Data":"65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968"} Nov 02 21:56:40.687770 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.687368 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:40.687770 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.687390 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" event={"ID":"186ef918-6b5a-4d10-b1e1-26e446e939b5","Type":"ContainerStarted","Data":"1bc3a3615102a7aabecd095c09f487e13c5b7af1283c5accb7eba7257498dea0"} Nov 02 21:56:40.689055 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.689026 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" event={"ID":"2736a738-51d9-4df9-bc6e-2b0257f98eb9","Type":"ContainerStarted","Data":"deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187"} Nov 02 21:56:40.689158 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.689072 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" event={"ID":"2736a738-51d9-4df9-bc6e-2b0257f98eb9","Type":"ContainerStarted","Data":"6e2a68318cd1dd9c45b8789326cc46241446fdc56b81498b66bef2087d495752"} Nov 02 21:56:40.689158 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.689110 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:40.809235 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.809149 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" podStartSLOduration=1.809128069 podStartE2EDuration="1.809128069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:40.798618125 +0000 UTC m=+1347.960713245" watchObservedRunningTime="2026-11-02 21:56:40.809128069 +0000 UTC m=+1347.971223191" Nov 02 21:56:40.915884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:40.915806 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" podStartSLOduration=1.9157860709999999 podStartE2EDuration="1.915786071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:40.910039497 +0000 UTC m=+1348.072134618" watchObservedRunningTime="2026-11-02 21:56:40.915786071 +0000 UTC m=+1348.077881191" Nov 02 21:56:41.609382 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:41.609350 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:41.609911 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:41.609886 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:41.610648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:41.610630 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 21:56:45.627378 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:45.627336 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:56:45.627886 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:45.627865 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:56:45.948868 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:45.948787 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7"] Nov 02 21:56:45.954760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:45.954736 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.017420 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.017384 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7"] Nov 02 21:56:46.017947 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.017916 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.018332 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.018307 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.018539 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.018517 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.018733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.018714 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.018932 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.018897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.019095 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.019079 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.020322 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.020301 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tn8x\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.099046 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.099020 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 21:56:46.099187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.099089 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 21:56:46.099187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.099090 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 21:56:46.099187 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.099128 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z7n66\"" Nov 02 21:56:46.124216 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124187 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.124367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.124367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.124367 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124359 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.124538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.124538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.124538 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.124512 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tn8x\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.125159 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.125114 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.126804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.126781 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.126907 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.126865 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.127346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.127324 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.127346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.127337 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.217737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.205580 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.243692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.219365 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tn8x\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hcgc7\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.273765 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.273733 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:46.762368 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:56:46.762331 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c153c9_cf4b_4292_85ec_71fd80ac9cab.slice/crio-ca18f20db367422bc08ab380315c75b2989958e1ece676b03145ad8b8381f979 WatchSource:0}: Error finding container ca18f20db367422bc08ab380315c75b2989958e1ece676b03145ad8b8381f979: Status 404 returned error can't find the container with id ca18f20db367422bc08ab380315c75b2989958e1ece676b03145ad8b8381f979 Nov 02 21:56:46.771258 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:46.771230 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7"] Nov 02 21:56:47.793655 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:47.793612 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" event={"ID":"d3c153c9-cf4b-4292-85ec-71fd80ac9cab","Type":"ContainerStarted","Data":"ca18f20db367422bc08ab380315c75b2989958e1ece676b03145ad8b8381f979"} Nov 02 21:56:51.702525 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:51.702491 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:56:51.702935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:51.702646 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:56:54.009372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:54.009325 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:54.009372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:54.009388 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:54.877250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:54.877200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" event={"ID":"d3c153c9-cf4b-4292-85ec-71fd80ac9cab","Type":"ContainerStarted","Data":"d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9"} Nov 02 21:56:54.877459 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:54.877298 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:55.094214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:55.094152 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" podStartSLOduration=2.851882117 podStartE2EDuration="10.094138633s" totalImagesPullingTime="7.242256516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:55.04126106 +0000 UTC m=+1362.203356179" watchObservedRunningTime="2026-11-02 21:56:55.094138633 +0000 UTC m=+1362.256233753" Nov 02 21:56:55.890061 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:55.890027 5091 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hcgc7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:56:55.890231 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:55.890119 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" podUID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:56:58.889355 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:58.889323 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 21:56:59.536382 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.536338 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx"] Nov 02 21:56:59.541158 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.541138 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.608112 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.608077 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx"] Nov 02 21:56:59.663760 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.662591 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.664054 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.664027 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d92g\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.664225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.664208 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.664460 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.664438 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.664733 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.664710 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.664914 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.664888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.665033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.665018 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.665219 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.665196 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.665342 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.665265 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767059 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767134 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767196 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767288 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767239 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d92g\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767515 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767492 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767602 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.767779 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.767712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.768152 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.768131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.769693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.769658 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.769928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.769910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.823830 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.823733 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.823966 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.823877 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d92g\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:56:59.855764 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:56:59.855738 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:57:00.290441 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:57:00.290409 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff668d6_1c08_4d97_882f_5d7e05372b88.slice/crio-8fd20a645fd52ee11e24d0bfa7b566af7811c2d66140084755e2355033e691d2 WatchSource:0}: Error finding container 8fd20a645fd52ee11e24d0bfa7b566af7811c2d66140084755e2355033e691d2: Status 404 returned error can't find the container with id 8fd20a645fd52ee11e24d0bfa7b566af7811c2d66140084755e2355033e691d2 Nov 02 21:57:00.292517 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:00.292492 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx"] Nov 02 21:57:00.293122 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:00.293091 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:00.293183 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:00.293174 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:00.293222 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:00.293201 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:00.948494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:00.948451 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" event={"ID":"6ff668d6-1c08-4d97-882f-5d7e05372b88","Type":"ContainerStarted","Data":"240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f"} Nov 02 21:57:00.948666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:00.948518 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" event={"ID":"6ff668d6-1c08-4d97-882f-5d7e05372b88","Type":"ContainerStarted","Data":"8fd20a645fd52ee11e24d0bfa7b566af7811c2d66140084755e2355033e691d2"} Nov 02 21:57:01.165004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:01.164954 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podStartSLOduration=2.164937777 podStartE2EDuration="2.164937777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:01.161986112 +0000 UTC m=+1368.324081235" watchObservedRunningTime="2026-11-02 21:57:01.164937777 +0000 UTC m=+1368.327032897" Nov 02 21:57:01.856022 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:01.855985 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:57:01.857636 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:01.857608 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:57:01.857767 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:01.857660 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 21:57:02.856818 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:02.856784 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:57:02.857215 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:02.856835 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 21:57:03.856983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:03.856943 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:57:03.857464 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:03.857000 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 21:57:04.856225 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:04.856187 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:57:04.856404 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:04.856244 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 21:57:05.856834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:05.856755 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:57:05.856834 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:05.856809 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 21:57:06.856885 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:06.856850 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 21:57:06.857297 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:06.856913 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 21:57:07.116295 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:57:07.116212 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:57:08.174503 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:08.174469 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:57:08.175951 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:08.175930 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:57:08.176298 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:08.176269 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 21:57:12.936073 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:12.935535 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:12.960809 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:12.960766 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:12.986819 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:12.986774 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.080696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.061006 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.144935 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.144010 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.360619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.360514 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.502694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.464769 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.502694 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.497439 5091 scope.go:122] "RemoveContainer" containerID="7ccda80562440704fc6bae441abed8c84ebc3a790a109e3ff17787d80e807e63" Nov 02 21:57:13.803708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.803646 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 21:57:13.804293 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.804263 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.826025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.825991 5091 scope.go:122] "RemoveContainer" containerID="e6ac5e89cd835aa3ce546fade4c9a7cd4f470d3dcfaf9facbd7fe0e2b8dff5fb" Nov 02 21:57:13.843730 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.830583 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Nov 02 21:57:13.882903 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.882647 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.943116 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.943068 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.944086 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.944061 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:57:13.972937 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.972896 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.984782 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:13.973546 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:57:14.024891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.024848 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 21:57:14.043695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.032361 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:57:14.190753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.188014 5091 scope.go:122] "RemoveContainer" containerID="d943e191a67ac044abece2437503a90501d68216f90e9fb71cc53429a8bbb930" Nov 02 21:57:14.190753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.189314 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.190753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.189739 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:57:14.211278 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.211240 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:57:14.211877 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.211853 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:57:14.505756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.361157 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.676203 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.523143 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 21:57:14.680241 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.680209 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:57:14.685692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.680435 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:57:14.711696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.523203 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 02 21:57:14.854755 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.829283 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:57:14.969698 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.867434 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:57:14.986843 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.986795 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:57:15.040695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.895377 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.040695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.955422 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.040695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.039191 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:57:15.040695 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.039370 5091 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.092875461s: [/var/lib/containers/storage/overlay/247a4b3fbad411ba9cf72eec39bddc184ed949631591adfb854fa947d4665888/diff /var/log/pods/openshift-ingress_e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv_592d0e95-5afb-4f73-8c3e-6594e08388c2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 02 21:57:15.067581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:14.891918 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.109753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.102797 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 21:57:15.200696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.189545 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.215900 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.215856 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.279859 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.279826 5091 scope.go:122] "RemoveContainer" containerID="2affd7c3413314098602aca66a708bef1e2fb0356d8750a9822673d936d80cf3" Nov 02 21:57:15.463702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.456012 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.602573 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.602538 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:57:15.692710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.685861 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.692710 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.687242 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 21:57:15.759415 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.759388 5091 scope.go:122] "RemoveContainer" containerID="071a3563a912a55ddfe4c9ddc26f584032809b35f7f5278d392f6b1090eac2ff" Nov 02 21:57:15.893423 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:15.891379 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:57:16.168664 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.168614 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.169618 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.169578 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.170984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.170958 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 21:57:16.190702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.175000 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:57:16.358071 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.196689 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 02 21:57:16.358808 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.196754 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 21:57:16.359262 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.297051 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:57:16.445914 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.445881 5091 scope.go:122] "RemoveContainer" containerID="05d5eb7696bd982c58b531da0739b2f73ebb1f8a53b678a7c148ee36261170a1" Nov 02 21:57:16.778750 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.750158 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.778750 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.750540 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.935692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:16.903350 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 21:57:17.183949 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.181766 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.183949 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.182183 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 21:57:17.183949 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.182473 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 21:57:17.393700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.386631 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.400844 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.400801 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 21:57:17.401398 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.401365 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 02 21:57:17.416255 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.416201 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 21:57:17.698059 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.696127 5091 scope.go:122] "RemoveContainer" containerID="9d7d42d8307aef00b574121e2b4476a5592e5b6fea7c6abe3db5d8dce9f799d9" Nov 02 21:57:17.730886 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.730846 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.752267 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.750648 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.761203 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.755976 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.787346 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.787312 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 21:57:17.787556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.787518 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 21:57:17.787943 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.787917 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 21:57:17.788229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.788087 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 21:57:17.788335 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.788313 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 21:57:17.788469 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.788454 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 21:57:17.788619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.788605 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 21:57:17.818693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.818340 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 21:57:17.818794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.818775 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 21:57:17.819610 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.819588 5091 scope.go:122] "RemoveContainer" containerID="b5c263486880341112ed7c1de0d6eceb88ef3a6a2659a8d5ad4d7dd844092fad" Nov 02 21:57:17.822220 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.822198 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 21:57:17.838324 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.838304 5091 scope.go:122] "RemoveContainer" containerID="ee5bee8ba6f4333de954033f6863ace3f9e4c25d94a3884d9e7300abbc2ee1d7" Nov 02 21:57:17.850709 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.850649 5091 scope.go:122] "RemoveContainer" containerID="e6f79d82bbe0126795b19e80d1d637816bbdbf4e56eed271a20eaf41e9b4fb2b" Nov 02 21:57:17.863714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.863651 5091 scope.go:122] "RemoveContainer" containerID="49645cc91813369acfeb3a0bb19880bbd5a5ca1d7c37c4ebf162e779ead4b8c9" Nov 02 21:57:17.873913 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.873892 5091 scope.go:122] "RemoveContainer" containerID="00cc9d8ef7eff175c895697026e7dd67f78a76de346da9905198c9d536583c58" Nov 02 21:57:17.884169 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:17.884147 5091 scope.go:122] "RemoveContainer" containerID="813767329ec08aa9ad332676c3bcb44fafcb6b422a713a061555387e46c98ec1" Nov 02 21:57:37.116960 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:57:37.116879 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:57:43.192621 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.192582 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8"] Nov 02 21:57:43.197644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.197618 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.252304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.252274 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8"] Nov 02 21:57:43.295568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.295532 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:57:43.295568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.295548 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:57:43.295568 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.295562 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:57:43.302444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.302416 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:57:43.308363 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.308328 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.308584 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.308564 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.308881 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.308856 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkcx\" (UniqueName: \"kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.410114 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.410071 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkcx\" (UniqueName: \"kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.410310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.410128 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.410310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.410277 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.418964 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.411462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.418964 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.415267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.466565 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.466498 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkcx\" (UniqueName: \"kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx\") pod \"maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.512138 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.512108 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:43.890037 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:57:43.890001 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1972ff7_3e43_44e4_8f0f_a621d76646a5.slice/crio-7dd5a17f41d47640278251602427092239bb1224d3af69dce96a05f5bcc8d545 WatchSource:0}: Error finding container 7dd5a17f41d47640278251602427092239bb1224d3af69dce96a05f5bcc8d545: Status 404 returned error can't find the container with id 7dd5a17f41d47640278251602427092239bb1224d3af69dce96a05f5bcc8d545 Nov 02 21:57:43.891445 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:43.891420 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8"] Nov 02 21:57:44.061928 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:44.061890 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" event={"ID":"f1972ff7-3e43-44e4-8f0f-a621d76646a5","Type":"ContainerStarted","Data":"7dd5a17f41d47640278251602427092239bb1224d3af69dce96a05f5bcc8d545"} Nov 02 21:57:44.982011 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:44.981974 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66"] Nov 02 21:57:44.986648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:44.986625 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034422 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034619 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034494 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034680 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034624 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034777 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034758 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034843 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034806 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034843 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034835 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034886 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpft5\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034918 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.034959 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.034952 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.045714 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.045662 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66"] Nov 02 21:57:45.073174 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.073134 5091 generic.go:356] "Generic (PLEG): container finished" podID="f1972ff7-3e43-44e4-8f0f-a621d76646a5" containerID="284aaba3fb45843d2cf0ec0c278767217574a2d76c2d1ad585e56d1c42bc25c2" exitCode=0 Nov 02 21:57:45.073349 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.073219 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" event={"ID":"f1972ff7-3e43-44e4-8f0f-a621d76646a5","Type":"ContainerDied","Data":"284aaba3fb45843d2cf0ec0c278767217574a2d76c2d1ad585e56d1c42bc25c2"} Nov 02 21:57:45.136808 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.136777 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137008 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.136820 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137008 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.136849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137008 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.136871 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpft5\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137008 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.136899 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137008 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.136929 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137015 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137260 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137485 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137296 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137585 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137564 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137659 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137604 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.137975 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.137954 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.139218 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.139194 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.139648 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.139625 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.193090 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.193060 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.193241 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.193146 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpft5\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.301209 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.301180 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:45.732703 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:57:45.732641 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc923486f_7415_4bf1_84f8_0dd19283d67a.slice/crio-c9a5ed75c646cec6d21fe54cc8645aec9e26625f4a5ede0f67594a519b9c652b WatchSource:0}: Error finding container c9a5ed75c646cec6d21fe54cc8645aec9e26625f4a5ede0f67594a519b9c652b: Status 404 returned error can't find the container with id c9a5ed75c646cec6d21fe54cc8645aec9e26625f4a5ede0f67594a519b9c652b Nov 02 21:57:45.734004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.733980 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66"] Nov 02 21:57:45.735422 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.735389 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:45.735497 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.735457 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:45.735497 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:45.735492 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:46.083732 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.083618 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" event={"ID":"c923486f-7415-4bf1-84f8-0dd19283d67a","Type":"ContainerStarted","Data":"fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f"} Nov 02 21:57:46.083732 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.083708 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" event={"ID":"c923486f-7415-4bf1-84f8-0dd19283d67a","Type":"ContainerStarted","Data":"c9a5ed75c646cec6d21fe54cc8645aec9e26625f4a5ede0f67594a519b9c652b"} Nov 02 21:57:46.282475 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.282449 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:46.301377 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.301347 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:46.303805 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.303769 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:46.303929 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.303835 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:46.357483 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.357388 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca\") pod \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " Nov 02 21:57:46.357483 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.357455 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca\") pod \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " Nov 02 21:57:46.357747 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.357531 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx\" (UniqueName: \"kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx\") pod \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\" (UID: \"f1972ff7-3e43-44e4-8f0f-a621d76646a5\") " Nov 02 21:57:46.357843 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.357821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca" pod "f1972ff7-3e43-44e4-8f0f-a621d76646a5" (UID: "f1972ff7-3e43-44e4-8f0f-a621d76646a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:46.358309 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.358287 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:57:46.359708 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.359681 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca" pod "f1972ff7-3e43-44e4-8f0f-a621d76646a5" (UID: "f1972ff7-3e43-44e4-8f0f-a621d76646a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:46.359819 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.359726 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx" pod "f1972ff7-3e43-44e4-8f0f-a621d76646a5" (UID: "f1972ff7-3e43-44e4-8f0f-a621d76646a5"). InnerVolumeSpecName "kube-api-access-hdkcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:46.406829 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.403514 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podStartSLOduration=2.403493459 podStartE2EDuration="2.403493459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:46.202126477 +0000 UTC m=+1413.364221598" watchObservedRunningTime="2026-11-02 21:57:46.403493459 +0000 UTC m=+1413.565588581" Nov 02 21:57:46.459940 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.459907 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkcx\" (UniqueName: \"kubernetes.io/projected/f1972ff7-3e43-44e4-8f0f-a621d76646a5-kube-api-access-hdkcx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:57:46.460095 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:46.459944 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1972ff7-3e43-44e4-8f0f-a621d76646a5-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:57:47.101161 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:47.101132 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" Nov 02 21:57:47.101569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:47.101175 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8" event={"ID":"f1972ff7-3e43-44e4-8f0f-a621d76646a5","Type":"ContainerDied","Data":"7dd5a17f41d47640278251602427092239bb1224d3af69dce96a05f5bcc8d545"} Nov 02 21:57:47.101569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:47.101223 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd5a17f41d47640278251602427092239bb1224d3af69dce96a05f5bcc8d545" Nov 02 21:57:47.301742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:47.301639 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:47.301742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:47.301716 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:48.302000 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:48.301966 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:48.302396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:48.302020 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:49.302631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:49.302344 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:49.302631 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:49.302407 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:50.137742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.137704 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn"] Nov 02 21:57:50.138472 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.138454 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f1972ff7-3e43-44e4-8f0f-a621d76646a5" containerName="revoke-keys" Nov 02 21:57:50.138558 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.138473 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1972ff7-3e43-44e4-8f0f-a621d76646a5" containerName="revoke-keys" Nov 02 21:57:50.138611 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.138601 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1972ff7-3e43-44e4-8f0f-a621d76646a5" containerName="revoke-keys" Nov 02 21:57:50.142632 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.142615 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.202098 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.201557 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn"] Nov 02 21:57:50.241380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.241348 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:57:50.241591 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.241463 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:57:50.241835 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.241812 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:57:50.245588 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.245565 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:57:50.302383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.302347 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:50.302534 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.302408 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:50.341310 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.341279 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.341692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.341405 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrvv\" (UniqueName: \"kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.341692 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.341469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.443290 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.443205 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.443434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.443309 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrvv\" (UniqueName: \"kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.443434 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.443351 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.443917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.443897 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.445778 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.445754 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.499681 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.499627 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrvv\" (UniqueName: \"kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv\") pod \"maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:50.756666 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:50.756588 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:51.189696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:51.183447 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn"] Nov 02 21:57:51.302257 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:51.302224 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:51.302448 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:51.302279 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:52.148586 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:52.148539 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" event={"ID":"230a4b2f-e006-4754-aa2d-d8a12cf27824","Type":"ContainerStarted","Data":"d60d217ef02fab451d4e4d95b09992799e74172bbc6f6c489438d5db279de94f"} Nov 02 21:57:52.149082 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:52.148601 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" event={"ID":"230a4b2f-e006-4754-aa2d-d8a12cf27824","Type":"ContainerStarted","Data":"592b9d86545c0514fe2eb285a617a40a2d2bbec4072ef9cfd9a90c2b4e68a477"} Nov 02 21:57:52.267135 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:52.267086 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" podStartSLOduration=2.267068846 podStartE2EDuration="2.267068846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:52.260622495 +0000 UTC m=+1419.422717616" watchObservedRunningTime="2026-11-02 21:57:52.267068846 +0000 UTC m=+1419.429163997" Nov 02 21:57:52.302237 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:52.302202 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 21:57:52.302401 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:52.302264 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 21:57:53.161612 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:53.161573 5091 generic.go:356] "Generic (PLEG): container finished" podID="230a4b2f-e006-4754-aa2d-d8a12cf27824" containerID="d60d217ef02fab451d4e4d95b09992799e74172bbc6f6c489438d5db279de94f" exitCode=0 Nov 02 21:57:53.162038 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:53.161660 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" event={"ID":"230a4b2f-e006-4754-aa2d-d8a12cf27824","Type":"ContainerDied","Data":"d60d217ef02fab451d4e4d95b09992799e74172bbc6f6c489438d5db279de94f"} Nov 02 21:57:53.307019 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:53.306982 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:53.307518 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:53.307499 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:53.308233 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:53.308210 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 21:57:54.369197 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.369169 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:54.414451 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.414417 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv\" (UniqueName: \"kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv\") pod \"230a4b2f-e006-4754-aa2d-d8a12cf27824\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " Nov 02 21:57:54.414623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.414482 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca\") pod \"230a4b2f-e006-4754-aa2d-d8a12cf27824\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " Nov 02 21:57:54.414623 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.414581 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca\") pod \"230a4b2f-e006-4754-aa2d-d8a12cf27824\" (UID: \"230a4b2f-e006-4754-aa2d-d8a12cf27824\") " Nov 02 21:57:54.414998 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.414968 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca" pod "230a4b2f-e006-4754-aa2d-d8a12cf27824" (UID: "230a4b2f-e006-4754-aa2d-d8a12cf27824"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:54.415173 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.415152 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:57:54.416745 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.416720 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv" pod "230a4b2f-e006-4754-aa2d-d8a12cf27824" (UID: "230a4b2f-e006-4754-aa2d-d8a12cf27824"). InnerVolumeSpecName "kube-api-access-gjrvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:54.416921 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.416744 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca" pod "230a4b2f-e006-4754-aa2d-d8a12cf27824" (UID: "230a4b2f-e006-4754-aa2d-d8a12cf27824"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:54.517354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.517304 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrvv\" (UniqueName: \"kubernetes.io/projected/230a4b2f-e006-4754-aa2d-d8a12cf27824-kube-api-access-gjrvv\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:57:54.517354 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:54.517348 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a4b2f-e006-4754-aa2d-d8a12cf27824-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:57:55.186043 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:55.186015 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" Nov 02 21:57:55.186559 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:55.186535 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn" event={"ID":"230a4b2f-e006-4754-aa2d-d8a12cf27824","Type":"ContainerDied","Data":"592b9d86545c0514fe2eb285a617a40a2d2bbec4072ef9cfd9a90c2b4e68a477"} Nov 02 21:57:55.186717 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:57:55.186572 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592b9d86545c0514fe2eb285a617a40a2d2bbec4072ef9cfd9a90c2b4e68a477" Nov 02 21:58:01.604652 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:01.604614 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt"] Nov 02 21:58:01.719813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:01.719777 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd"] Nov 02 21:58:06.605941 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:06.605891 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" podUID="70a3809c-9391-4531-9285-c9233b10d30a" containerName="payload-processing" containerID="cri-o://ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35" gracePeriod=25 Nov 02 21:58:06.720545 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:06.720477 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" podUID="fb5e11da-1a68-434b-a084-709556e587f9" containerName="payload-pre-processing" containerID="cri-o://c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e" gracePeriod=25 Nov 02 21:58:06.983739 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:06.983712 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:58:07.031375 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.030965 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume\") pod \"70a3809c-9391-4531-9285-c9233b10d30a\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " Nov 02 21:58:07.031753 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.031350 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z\" (UniqueName: \"kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z\") pod \"70a3809c-9391-4531-9285-c9233b10d30a\" (UID: \"70a3809c-9391-4531-9285-c9233b10d30a\") " Nov 02 21:58:07.033166 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.033140 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume" pod "70a3809c-9391-4531-9285-c9233b10d30a" (UID: "70a3809c-9391-4531-9285-c9233b10d30a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:07.044469 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.044421 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z" pod "70a3809c-9391-4531-9285-c9233b10d30a" (UID: "70a3809c-9391-4531-9285-c9233b10d30a"). InnerVolumeSpecName "kube-api-access-mmn2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:07.127214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.126833 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:58:07.127214 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.126899 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:58:07.135412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.135355 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmn2z\" (UniqueName: \"kubernetes.io/projected/70a3809c-9391-4531-9285-c9233b10d30a-kube-api-access-mmn2z\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:07.135412 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.135388 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a3809c-9391-4531-9285-c9233b10d30a-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:07.137620 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:58:07.137559 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:58:07.145593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.145565 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:58:07.236721 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.236638 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w\" (UniqueName: \"kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w\") pod \"fb5e11da-1a68-434b-a084-709556e587f9\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " Nov 02 21:58:07.236867 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.236851 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume\") pod \"fb5e11da-1a68-434b-a084-709556e587f9\" (UID: \"fb5e11da-1a68-434b-a084-709556e587f9\") " Nov 02 21:58:07.237242 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.237214 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume" pod "fb5e11da-1a68-434b-a084-709556e587f9" (UID: "fb5e11da-1a68-434b-a084-709556e587f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:07.237383 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.237365 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5e11da-1a68-434b-a084-709556e587f9-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:07.238816 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.238796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w" pod "fb5e11da-1a68-434b-a084-709556e587f9" (UID: "fb5e11da-1a68-434b-a084-709556e587f9"). InnerVolumeSpecName "kube-api-access-7jm6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:07.339304 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.339262 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jm6w\" (UniqueName: \"kubernetes.io/projected/fb5e11da-1a68-434b-a084-709556e587f9-kube-api-access-7jm6w\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:07.378386 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.378351 5091 generic.go:356] "Generic (PLEG): container finished" podID="fb5e11da-1a68-434b-a084-709556e587f9" containerID="c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e" exitCode=0 Nov 02 21:58:07.378569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.378404 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" Nov 02 21:58:07.378569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.378441 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" event={"ID":"fb5e11da-1a68-434b-a084-709556e587f9","Type":"ContainerDied","Data":"c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e"} Nov 02 21:58:07.378569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.378508 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd" event={"ID":"fb5e11da-1a68-434b-a084-709556e587f9","Type":"ContainerDied","Data":"1bca548fc803714bd5f628c7962bbfcc26f5d3955ad18b3d277098f178aa6fc8"} Nov 02 21:58:07.378569 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.378541 5091 scope.go:122] "RemoveContainer" containerID="c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e" Nov 02 21:58:07.380107 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.380086 5091 generic.go:356] "Generic (PLEG): container finished" podID="70a3809c-9391-4531-9285-c9233b10d30a" containerID="ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35" exitCode=0 Nov 02 21:58:07.380188 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.380129 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" Nov 02 21:58:07.380188 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.380132 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" event={"ID":"70a3809c-9391-4531-9285-c9233b10d30a","Type":"ContainerDied","Data":"ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35"} Nov 02 21:58:07.380305 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.380187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt" event={"ID":"70a3809c-9391-4531-9285-c9233b10d30a","Type":"ContainerDied","Data":"fa2266be391119bb68672df7f900d86929a39fd80e0c0fcaf13c0c23a3795c62"} Nov 02 21:58:07.391351 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.391332 5091 scope.go:122] "RemoveContainer" containerID="c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e" Nov 02 21:58:07.391645 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:58:07.391617 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e\": container with ID starting with c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e not found: ID does not exist" containerID="c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e" Nov 02 21:58:07.391759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.391655 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e"} err="failed to get container status \"c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e\": rpc error: code = NotFound desc = could not find container \"c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e\": container with ID starting with c2638a2bd336a74e098fb609ab0b51ed370b0a3cd015c8a4ea85389337a1c41e not found: ID does not exist" Nov 02 21:58:07.391759 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.391721 5091 scope.go:122] "RemoveContainer" containerID="ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35" Nov 02 21:58:07.404642 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.404625 5091 scope.go:122] "RemoveContainer" containerID="ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35" Nov 02 21:58:07.405001 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:58:07.404972 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35\": container with ID starting with ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35 not found: ID does not exist" containerID="ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35" Nov 02 21:58:07.405105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.405009 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35"} err="failed to get container status \"ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35\": rpc error: code = NotFound desc = could not find container \"ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35\": container with ID starting with ea1c2edddd74eb5a002900f79bab2e580078db074877e683eee534dcd21cef35 not found: ID does not exist" Nov 02 21:58:07.547843 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.547756 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt"] Nov 02 21:58:07.556783 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.556756 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a16a02cb-8bcd4895c-pt8nt"] Nov 02 21:58:07.658984 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.658951 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd"] Nov 02 21:58:07.664918 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:07.664891 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a16a02cb-59bdb95dc9-kctpd"] Nov 02 21:58:08.981372 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:08.981337 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a3809c-9391-4531-9285-c9233b10d30a" path="/var/lib/kubelet/pods/70a3809c-9391-4531-9285-c9233b10d30a/volumes" Nov 02 21:58:08.985328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:08.981987 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5e11da-1a68-434b-a084-709556e587f9" path="/var/lib/kubelet/pods/fb5e11da-1a68-434b-a084-709556e587f9/volumes" Nov 02 21:58:09.410720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:09.410655 5091 generic.go:356] "Generic (PLEG): container finished" podID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerID="b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712" exitCode=0 Nov 02 21:58:09.410720 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:09.410699 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" event={"ID":"544748cb-be52-4ce3-8772-78998f8b5fe8","Type":"ContainerDied","Data":"b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712"} Nov 02 21:58:09.411176 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:09.411158 5091 scope.go:122] "RemoveContainer" containerID="b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712" Nov 02 21:58:09.412589 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:09.412563 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:09.412643 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:09.412627 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:09.412712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:09.412659 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:10.424035 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:10.423998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" event={"ID":"544748cb-be52-4ce3-8772-78998f8b5fe8","Type":"ContainerStarted","Data":"ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365"} Nov 02 21:58:10.960757 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:10.960722 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:58:10.960939 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:10.960921 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 21:58:10.960989 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:10.960974 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 21:58:11.960889 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:11.960855 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 21:58:11.961314 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:11.960914 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 21:58:12.961474 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:12.961434 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 21:58:12.961916 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:12.961499 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 21:58:13.961044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:13.961007 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 21:58:13.961222 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:13.961071 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 21:58:14.961144 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:14.961110 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 21:58:14.961566 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:14.961171 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 21:58:15.965203 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:15.965174 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:58:15.965742 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:15.965563 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:58:15.966228 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:15.966208 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:58:18.026942 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:18.026909 5091 scope.go:122] "RemoveContainer" containerID="483867d7a73992022072c8867a83fa5834514b47985b2e2984314ab9b202e939" Nov 02 21:58:18.038368 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:18.038345 5091 scope.go:122] "RemoveContainer" containerID="e89abf3066653e40e15ae33da67432187b97652ad4b4a2eb68779b4929486f9d" Nov 02 21:58:18.048150 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:18.048119 5091 scope.go:122] "RemoveContainer" containerID="a563a26487d5fd0d082661bc2356cac14bbca3bdd831c322c00c21b3333a8656" Nov 02 21:58:18.058052 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:18.058034 5091 scope.go:122] "RemoveContainer" containerID="8e83e5fdad603c16cc7cbe1ef0335b19a35b6edc21b02c3f4f738bd357181030" Nov 02 21:58:24.885756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.885715 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj"] Nov 02 21:58:24.886641 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886617 5091 cpu_manager.go:414] "containerMap: removing container" podUID="70a3809c-9391-4531-9285-c9233b10d30a" containerName="payload-processing" Nov 02 21:58:24.886773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886642 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a3809c-9391-4531-9285-c9233b10d30a" containerName="payload-processing" Nov 02 21:58:24.886773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886701 5091 cpu_manager.go:414] "containerMap: removing container" podUID="230a4b2f-e006-4754-aa2d-d8a12cf27824" containerName="revoke-keys" Nov 02 21:58:24.886773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886709 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230a4b2f-e006-4754-aa2d-d8a12cf27824" containerName="revoke-keys" Nov 02 21:58:24.886773 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886770 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fb5e11da-1a68-434b-a084-709556e587f9" containerName="payload-pre-processing" Nov 02 21:58:24.886978 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886779 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5e11da-1a68-434b-a084-709556e587f9" containerName="payload-pre-processing" Nov 02 21:58:24.887034 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.886984 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a3809c-9391-4531-9285-c9233b10d30a" containerName="payload-processing" Nov 02 21:58:24.887034 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.887005 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="230a4b2f-e006-4754-aa2d-d8a12cf27824" containerName="revoke-keys" Nov 02 21:58:24.887034 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.887020 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5e11da-1a68-434b-a084-709556e587f9" containerName="payload-pre-processing" Nov 02 21:58:24.890820 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.890798 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:24.948875 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.948845 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj"] Nov 02 21:58:24.990716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.990687 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:58:24.990716 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.990718 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:58:24.992154 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.992131 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:58:24.992243 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.992132 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:58:24.992799 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.992775 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:24.992964 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.992945 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lvh\" (UniqueName: \"kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:24.993071 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:24.993053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.094040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.094013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.094174 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.094150 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lvh\" (UniqueName: \"kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.094265 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.094242 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.094933 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.094908 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.096438 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.096419 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.152020 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.151957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lvh\" (UniqueName: \"kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh\") pod \"maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.204657 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.204627 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:25.593482 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:25.593454 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj"] Nov 02 21:58:25.595733 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:58:25.595704 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc78bd87_b77c_4f98_b4cd_81c9d1124bec.slice/crio-8258111074fa3c67bbcab4de80d08af2c093ea440830c0f61717662b5fb34841 WatchSource:0}: Error finding container 8258111074fa3c67bbcab4de80d08af2c093ea440830c0f61717662b5fb34841: Status 404 returned error can't find the container with id 8258111074fa3c67bbcab4de80d08af2c093ea440830c0f61717662b5fb34841 Nov 02 21:58:26.595064 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:26.595031 5091 generic.go:356] "Generic (PLEG): container finished" podID="bc78bd87-b77c-4f98-b4cd-81c9d1124bec" containerID="e2160971d6520ab7c6ebf1611e1b35cacf9fe40726f587a066615940618acda7" exitCode=0 Nov 02 21:58:26.595444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:26.595080 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" event={"ID":"bc78bd87-b77c-4f98-b4cd-81c9d1124bec","Type":"ContainerDied","Data":"e2160971d6520ab7c6ebf1611e1b35cacf9fe40726f587a066615940618acda7"} Nov 02 21:58:26.595444 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:26.595130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" event={"ID":"bc78bd87-b77c-4f98-b4cd-81c9d1124bec","Type":"ContainerStarted","Data":"8258111074fa3c67bbcab4de80d08af2c093ea440830c0f61717662b5fb34841"} Nov 02 21:58:27.790453 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.790419 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:27.935682 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.935633 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca\") pod \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " Nov 02 21:58:27.935927 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.935904 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh\" (UniqueName: \"kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh\") pod \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " Nov 02 21:58:27.935983 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.935962 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca" pod "bc78bd87-b77c-4f98-b4cd-81c9d1124bec" (UID: "bc78bd87-b77c-4f98-b4cd-81c9d1124bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:27.936139 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.936122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca\") pod \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\" (UID: \"bc78bd87-b77c-4f98-b4cd-81c9d1124bec\") " Nov 02 21:58:27.936786 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.936761 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:27.938168 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.938136 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca" pod "bc78bd87-b77c-4f98-b4cd-81c9d1124bec" (UID: "bc78bd87-b77c-4f98-b4cd-81c9d1124bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:27.938263 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:27.938180 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh" pod "bc78bd87-b77c-4f98-b4cd-81c9d1124bec" (UID: "bc78bd87-b77c-4f98-b4cd-81c9d1124bec"). InnerVolumeSpecName "kube-api-access-l4lvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:28.038380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:28.038346 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4lvh\" (UniqueName: \"kubernetes.io/projected/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-kube-api-access-l4lvh\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:28.038380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:28.038378 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc78bd87-b77c-4f98-b4cd-81c9d1124bec-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:28.613813 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:28.613782 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" Nov 02 21:58:28.613995 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:28.613820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj" event={"ID":"bc78bd87-b77c-4f98-b4cd-81c9d1124bec","Type":"ContainerDied","Data":"8258111074fa3c67bbcab4de80d08af2c093ea440830c0f61717662b5fb34841"} Nov 02 21:58:28.613995 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:28.613868 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8258111074fa3c67bbcab4de80d08af2c093ea440830c0f61717662b5fb34841" Nov 02 21:58:34.309596 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:34.309556 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8"] Nov 02 21:58:34.363416 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:34.363381 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c2719c1-ws9x8"] Nov 02 21:58:34.983625 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:34.983588 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1972ff7-3e43-44e4-8f0f-a621d76646a5" path="/var/lib/kubelet/pods/f1972ff7-3e43-44e4-8f0f-a621d76646a5/volumes" Nov 02 21:58:36.933593 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:36.933563 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl"] Nov 02 21:58:37.051895 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:37.051862 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc"] Nov 02 21:58:37.120856 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:58:37.120820 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:58:40.943618 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:40.943577 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn"] Nov 02 21:58:40.962025 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:40.961978 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a16a02cb-7bzbn"] Nov 02 21:58:40.988359 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:40.988315 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230a4b2f-e006-4754-aa2d-d8a12cf27824" path="/var/lib/kubelet/pods/230a4b2f-e006-4754-aa2d-d8a12cf27824/volumes" Nov 02 21:58:41.934244 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:41.934205 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" podUID="186ef918-6b5a-4d10-b1e1-26e446e939b5" containerName="payload-processing" containerID="cri-o://65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968" gracePeriod=25 Nov 02 21:58:42.052366 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.052150 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" podUID="2736a738-51d9-4df9-bc6e-2b0257f98eb9" containerName="payload-pre-processing" containerID="cri-o://deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187" gracePeriod=25 Nov 02 21:58:42.449693 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.448455 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:58:42.605328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.602132 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft\" (UniqueName: \"kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft\") pod \"186ef918-6b5a-4d10-b1e1-26e446e939b5\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " Nov 02 21:58:42.605328 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.602276 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume\") pod \"186ef918-6b5a-4d10-b1e1-26e446e939b5\" (UID: \"186ef918-6b5a-4d10-b1e1-26e446e939b5\") " Nov 02 21:58:42.606151 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.605791 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume" pod "186ef918-6b5a-4d10-b1e1-26e446e939b5" (UID: "186ef918-6b5a-4d10-b1e1-26e446e939b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:42.622789 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.622650 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft" pod "186ef918-6b5a-4d10-b1e1-26e446e939b5" (UID: "186ef918-6b5a-4d10-b1e1-26e446e939b5"). InnerVolumeSpecName "kube-api-access-97kft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:42.700378 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.700340 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:58:42.709030 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.708987 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kft\" (UniqueName: \"kubernetes.io/projected/186ef918-6b5a-4d10-b1e1-26e446e939b5-kube-api-access-97kft\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:42.709339 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.709154 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/186ef918-6b5a-4d10-b1e1-26e446e939b5-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:42.810177 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.810138 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6\") pod \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " Nov 02 21:58:42.810384 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.810358 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume\") pod \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\" (UID: \"2736a738-51d9-4df9-bc6e-2b0257f98eb9\") " Nov 02 21:58:42.810776 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.810746 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume" pod "2736a738-51d9-4df9-bc6e-2b0257f98eb9" (UID: "2736a738-51d9-4df9-bc6e-2b0257f98eb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:42.811277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.811248 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2736a738-51d9-4df9-bc6e-2b0257f98eb9-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:42.812512 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.812483 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6" pod "2736a738-51d9-4df9-bc6e-2b0257f98eb9" (UID: "2736a738-51d9-4df9-bc6e-2b0257f98eb9"). InnerVolumeSpecName "kube-api-access-pktn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:42.907941 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.907905 5091 generic.go:356] "Generic (PLEG): container finished" podID="2736a738-51d9-4df9-bc6e-2b0257f98eb9" containerID="deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187" exitCode=0 Nov 02 21:58:42.908149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.907970 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" Nov 02 21:58:42.908149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.907973 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" event={"ID":"2736a738-51d9-4df9-bc6e-2b0257f98eb9","Type":"ContainerDied","Data":"deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187"} Nov 02 21:58:42.908149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.908053 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc" event={"ID":"2736a738-51d9-4df9-bc6e-2b0257f98eb9","Type":"ContainerDied","Data":"6e2a68318cd1dd9c45b8789326cc46241446fdc56b81498b66bef2087d495752"} Nov 02 21:58:42.908149 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.908085 5091 scope.go:122] "RemoveContainer" containerID="deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187" Nov 02 21:58:42.911229 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.911207 5091 generic.go:356] "Generic (PLEG): container finished" podID="186ef918-6b5a-4d10-b1e1-26e446e939b5" containerID="65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968" exitCode=0 Nov 02 21:58:42.911458 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.911431 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" event={"ID":"186ef918-6b5a-4d10-b1e1-26e446e939b5","Type":"ContainerDied","Data":"65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968"} Nov 02 21:58:42.911741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.911484 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" event={"ID":"186ef918-6b5a-4d10-b1e1-26e446e939b5","Type":"ContainerDied","Data":"1bc3a3615102a7aabecd095c09f487e13c5b7af1283c5accb7eba7257498dea0"} Nov 02 21:58:42.911741 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.911576 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl" Nov 02 21:58:42.915880 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.915854 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktn6\" (UniqueName: \"kubernetes.io/projected/2736a738-51d9-4df9-bc6e-2b0257f98eb9-kube-api-access-pktn6\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:42.923878 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.923855 5091 scope.go:122] "RemoveContainer" containerID="deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187" Nov 02 21:58:42.924464 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:58:42.924279 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187\": container with ID starting with deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187 not found: ID does not exist" containerID="deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187" Nov 02 21:58:42.924464 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.924326 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187"} err="failed to get container status \"deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187\": rpc error: code = NotFound desc = could not find container \"deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187\": container with ID starting with deee14f9a565d2ee71fcc522624f11202a692287a5f51f1d983b9fb47b73b187 not found: ID does not exist" Nov 02 21:58:42.924464 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.924352 5091 scope.go:122] "RemoveContainer" containerID="65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968" Nov 02 21:58:42.936956 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.936813 5091 scope.go:122] "RemoveContainer" containerID="65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968" Nov 02 21:58:42.937190 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:58:42.937167 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968\": container with ID starting with 65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968 not found: ID does not exist" containerID="65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968" Nov 02 21:58:42.937277 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:42.937197 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968"} err="failed to get container status \"65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968\": rpc error: code = NotFound desc = could not find container \"65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968\": container with ID starting with 65bda9f33f19457535bea158ac895fffae8a60a289072993450d7b04d4788968 not found: ID does not exist" Nov 02 21:58:43.073851 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.073817 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl"] Nov 02 21:58:43.079369 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.079341 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fb2343fe-846798d7f7-6mnzl"] Nov 02 21:58:43.188837 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.188762 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc"] Nov 02 21:58:43.197532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.197504 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb2343fe-79c4fbfb55-sw5lc"] Nov 02 21:58:43.206427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.206400 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs"] Nov 02 21:58:43.207478 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207458 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2736a738-51d9-4df9-bc6e-2b0257f98eb9" containerName="payload-pre-processing" Nov 02 21:58:43.207532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207482 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2736a738-51d9-4df9-bc6e-2b0257f98eb9" containerName="payload-pre-processing" Nov 02 21:58:43.207532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207503 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bc78bd87-b77c-4f98-b4cd-81c9d1124bec" containerName="revoke-keys" Nov 02 21:58:43.207532 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207511 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc78bd87-b77c-4f98-b4cd-81c9d1124bec" containerName="revoke-keys" Nov 02 21:58:43.207632 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207534 5091 cpu_manager.go:414] "containerMap: removing container" podUID="186ef918-6b5a-4d10-b1e1-26e446e939b5" containerName="payload-processing" Nov 02 21:58:43.207632 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207541 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186ef918-6b5a-4d10-b1e1-26e446e939b5" containerName="payload-processing" Nov 02 21:58:43.207756 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207741 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2736a738-51d9-4df9-bc6e-2b0257f98eb9" containerName="payload-pre-processing" Nov 02 21:58:43.207789 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207761 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc78bd87-b77c-4f98-b4cd-81c9d1124bec" containerName="revoke-keys" Nov 02 21:58:43.207789 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.207774 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="186ef918-6b5a-4d10-b1e1-26e446e939b5" containerName="payload-processing" Nov 02 21:58:43.213523 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.213501 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.265167 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.265138 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs"] Nov 02 21:58:43.265737 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.265700 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:58:43.265862 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.265789 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:58:43.274923 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.274859 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:58:43.275105 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.275088 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:58:43.322287 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.322255 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j5w\" (UniqueName: \"kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.322427 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.322393 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.322529 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.322443 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.424320 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.424284 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j5w\" (UniqueName: \"kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.424507 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.424408 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.424507 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.424470 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.425104 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.425077 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.426871 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.426852 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.480858 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.480797 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j5w\" (UniqueName: \"kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w\") pod \"maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.529811 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.529771 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:43.919042 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:43.918665 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs"] Nov 02 21:58:44.054190 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.054160 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl"] Nov 02 21:58:44.061463 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.061435 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d227f2f-mszvl"] Nov 02 21:58:44.943537 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.943497 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" event={"ID":"01ac0adf-5e90-4795-8bfa-a6ff46d936a5","Type":"ContainerStarted","Data":"cc659fb48632ec6a2d61aa26c0e2b2d0cd32c34c1df2353b1124f348fb63ff2b"} Nov 02 21:58:44.944120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.943572 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" event={"ID":"01ac0adf-5e90-4795-8bfa-a6ff46d936a5","Type":"ContainerStarted","Data":"938775d039eb56ea4ada2f48e73750e6e169b3c4e6f539da4056bdc27594f96e"} Nov 02 21:58:44.989122 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.989091 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00af4ae4-38f5-47b9-b914-3f11179abb86" path="/var/lib/kubelet/pods/00af4ae4-38f5-47b9-b914-3f11179abb86/volumes" Nov 02 21:58:44.989873 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.989849 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186ef918-6b5a-4d10-b1e1-26e446e939b5" path="/var/lib/kubelet/pods/186ef918-6b5a-4d10-b1e1-26e446e939b5/volumes" Nov 02 21:58:44.990413 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:44.990388 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2736a738-51d9-4df9-bc6e-2b0257f98eb9" path="/var/lib/kubelet/pods/2736a738-51d9-4df9-bc6e-2b0257f98eb9/volumes" Nov 02 21:58:45.071636 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:45.066413 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" podStartSLOduration=2.066340604 podStartE2EDuration="2.066340604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:45.057281346 +0000 UTC m=+1472.219376460" watchObservedRunningTime="2026-11-02 21:58:45.066340604 +0000 UTC m=+1472.228435725" Nov 02 21:58:45.954033 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:45.953987 5091 generic.go:356] "Generic (PLEG): container finished" podID="01ac0adf-5e90-4795-8bfa-a6ff46d936a5" containerID="cc659fb48632ec6a2d61aa26c0e2b2d0cd32c34c1df2353b1124f348fb63ff2b" exitCode=0 Nov 02 21:58:45.954506 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:45.954135 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" event={"ID":"01ac0adf-5e90-4795-8bfa-a6ff46d936a5","Type":"ContainerDied","Data":"cc659fb48632ec6a2d61aa26c0e2b2d0cd32c34c1df2353b1124f348fb63ff2b"} Nov 02 21:58:47.160527 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.160495 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:58:47.185644 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.185604 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w\" (UniqueName: \"kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w\") pod \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " Nov 02 21:58:47.185882 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.185685 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca\") pod \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " Nov 02 21:58:47.186013 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.185993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca\") pod \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\" (UID: \"01ac0adf-5e90-4795-8bfa-a6ff46d936a5\") " Nov 02 21:58:47.186199 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.186172 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca" pod "01ac0adf-5e90-4795-8bfa-a6ff46d936a5" (UID: "01ac0adf-5e90-4795-8bfa-a6ff46d936a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:47.186730 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.186657 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:47.188254 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.188225 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca" pod "01ac0adf-5e90-4795-8bfa-a6ff46d936a5" (UID: "01ac0adf-5e90-4795-8bfa-a6ff46d936a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:47.188406 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.188392 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w" pod "01ac0adf-5e90-4795-8bfa-a6ff46d936a5" (UID: "01ac0adf-5e90-4795-8bfa-a6ff46d936a5"). InnerVolumeSpecName "kube-api-access-g9j5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:47.288792 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.288704 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9j5w\" (UniqueName: \"kubernetes.io/projected/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-kube-api-access-g9j5w\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:47.288792 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.288743 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ac0adf-5e90-4795-8bfa-a6ff46d936a5-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:47.990702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.990167 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" event={"ID":"01ac0adf-5e90-4795-8bfa-a6ff46d936a5","Type":"ContainerDied","Data":"938775d039eb56ea4ada2f48e73750e6e169b3c4e6f539da4056bdc27594f96e"} Nov 02 21:58:47.990702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.990231 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="938775d039eb56ea4ada2f48e73750e6e169b3c4e6f539da4056bdc27594f96e" Nov 02 21:58:47.990702 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:58:47.990301 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs" Nov 02 21:59:03.047498 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:03.047422 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w"] Nov 02 21:59:03.096754 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:03.096724 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1035b95-ctj9w"] Nov 02 21:59:04.978070 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:04.978036 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fedef5dc-cec8-498d-b928-862f7a0ef23d" path="/var/lib/kubelet/pods/fedef5dc-cec8-498d-b928-862f7a0ef23d/volumes" Nov 02 21:59:07.114485 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:59:07.114452 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:59:11.557211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:11.557179 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj"] Nov 02 21:59:11.610122 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:11.610082 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb2343fe-t6ctj"] Nov 02 21:59:11.618040 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:11.618011 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr"] Nov 02 21:59:11.618220 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:11.618199 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" containerID="cri-o://ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365" gracePeriod=30 Nov 02 21:59:12.978093 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:12.978059 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc78bd87-b77c-4f98-b4cd-81c9d1124bec" path="/var/lib/kubelet/pods/bc78bd87-b77c-4f98-b4cd-81c9d1124bec/volumes" Nov 02 21:59:13.231926 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:13.231850 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:59:13.237016 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:13.236991 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 21:59:15.048800 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:15.048765 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb"] Nov 02 21:59:15.098144 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:15.098110 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10dc4518-p6klb"] Nov 02 21:59:15.968400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:15.968362 5091 patch_prober.go:29] interesting pod/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:59:15.968603 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:15.968429 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:59:16.939791 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:16.939756 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:59:16.979549 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:16.979517 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efddd336-a95e-4fea-be2c-a2bcb390bbec" path="/var/lib/kubelet/pods/efddd336-a95e-4fea-be2c-a2bcb390bbec/volumes" Nov 02 21:59:17.043117 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043083 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.043291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043192 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.043341 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043287 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.043341 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.043435 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043368 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.043494 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043434 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.043550 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043495 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:17.043812 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:17.043940 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043892 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.044006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043937 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:17.044006 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.043982 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.044118 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044067 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w\") pod \"544748cb-be52-4ce3-8772-78998f8b5fe8\" (UID: \"544748cb-be52-4ce3-8772-78998f8b5fe8\") " Nov 02 21:59:17.044184 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044134 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:17.044396 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044373 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:17.044891 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044868 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.044973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044896 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.044973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044910 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544748cb-be52-4ce3-8772-78998f8b5fe8-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.044973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044927 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.044973 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.044941 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.045553 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.045532 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:17.045723 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.045707 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:17.045798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.045781 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:17.046185 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.046165 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w" pod "544748cb-be52-4ce3-8772-78998f8b5fe8" (UID: "544748cb-be52-4ce3-8772-78998f8b5fe8"). InnerVolumeSpecName "kube-api-access-96d2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:17.147311 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.147271 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96d2w\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-kube-api-access-96d2w\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.147311 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.147314 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.147560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.147329 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.147560 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.147343 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544748cb-be52-4ce3-8772-78998f8b5fe8-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:17.265325 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.265235 5091 generic.go:356] "Generic (PLEG): container finished" podID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerID="ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365" exitCode=0 Nov 02 21:59:17.265325 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.265290 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" event={"ID":"544748cb-be52-4ce3-8772-78998f8b5fe8","Type":"ContainerDied","Data":"ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365"} Nov 02 21:59:17.265581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.265356 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" event={"ID":"544748cb-be52-4ce3-8772-78998f8b5fe8","Type":"ContainerDied","Data":"5f91898f8adc40b0a1651fcd502b1c249631a14f6af995abf8671964fa9f2726"} Nov 02 21:59:17.265581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.265297 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr" Nov 02 21:59:17.265581 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.265385 5091 scope.go:122] "RemoveContainer" containerID="ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365" Nov 02 21:59:17.277840 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.277820 5091 scope.go:122] "RemoveContainer" containerID="b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712" Nov 02 21:59:17.289004 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.288985 5091 scope.go:122] "RemoveContainer" containerID="ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365" Nov 02 21:59:17.289245 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:59:17.289227 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365\": container with ID starting with ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365 not found: ID does not exist" containerID="ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365" Nov 02 21:59:17.289295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.289252 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365"} err="failed to get container status \"ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365\": rpc error: code = NotFound desc = could not find container \"ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365\": container with ID starting with ace64bdfacb9a145433fdb0cddd3557fa92ffcb68d9ea0afffc7d4e1222a7365 not found: ID does not exist" Nov 02 21:59:17.289295 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.289270 5091 scope.go:122] "RemoveContainer" containerID="b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712" Nov 02 21:59:17.289500 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:59:17.289477 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712\": container with ID starting with b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712 not found: ID does not exist" containerID="b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712" Nov 02 21:59:17.289556 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.289509 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712"} err="failed to get container status \"b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712\": rpc error: code = NotFound desc = could not find container \"b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712\": container with ID starting with b208babdcab656349d7bd9579ae38139ca569f3c0ba40df5960ce8fb10fa6712 not found: ID does not exist" Nov 02 21:59:17.445281 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.445236 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr"] Nov 02 21:59:17.445769 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.445742 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fb2343fe-openshift-default-6bd5bd9df-p7jfr"] Nov 02 21:59:17.853712 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.849134 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs"] Nov 02 21:59:17.855269 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:17.855240 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-00052a2d-fjvhs"] Nov 02 21:59:18.979199 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:18.979165 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ac0adf-5e90-4795-8bfa-a6ff46d936a5" path="/var/lib/kubelet/pods/01ac0adf-5e90-4795-8bfa-a6ff46d936a5/volumes" Nov 02 21:59:18.979650 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:18.979634 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" path="/var/lib/kubelet/pods/544748cb-be52-4ce3-8772-78998f8b5fe8/volumes" Nov 02 21:59:20.046713 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:20.046665 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh"] Nov 02 21:59:20.096153 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:20.096126 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3641a89f-69dnh"] Nov 02 21:59:20.978241 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:20.978207 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfbc9c2-d9ba-4951-ac7b-7d4010549361" path="/var/lib/kubelet/pods/3bfbc9c2-d9ba-4951-ac7b-7d4010549361/volumes" Nov 02 21:59:31.047804 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:31.047770 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk"] Nov 02 21:59:31.097204 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:31.097171 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20e6269-rqvqk"] Nov 02 21:59:32.988317 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:32.988285 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187edd17-e602-4707-8a36-7ee14162b6b9" path="/var/lib/kubelet/pods/187edd17-e602-4707-8a36-7ee14162b6b9/volumes" Nov 02 21:59:37.136441 ip-10-0-128-36 kubenswrapper[5091]: E1102 21:59:37.136400 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 21:59:40.047156 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:40.047122 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz"] Nov 02 21:59:40.096044 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:40.096013 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57de505-n7rlz"] Nov 02 21:59:40.978353 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:40.978317 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c353267-d739-4ffd-b939-57ce1baf09bf" path="/var/lib/kubelet/pods/3c353267-d739-4ffd-b939-57ce1baf09bf/volumes" Nov 02 21:59:47.049705 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:47.049649 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b"] Nov 02 21:59:47.069319 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:47.069287 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bff44a-wcv6b"] Nov 02 21:59:48.978250 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:48.978214 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3763db08-91c7-4b93-b380-30c9e5bd0248" path="/var/lib/kubelet/pods/3763db08-91c7-4b93-b380-30c9e5bd0248/volumes" Nov 02 21:59:51.045696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:51.045649 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j"] Nov 02 21:59:51.094987 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:51.094948 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1819b8d4-ccp4j"] Nov 02 21:59:52.978144 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:52.978114 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acee79a2-7515-4de1-b7fa-f181ac2e9a14" path="/var/lib/kubelet/pods/acee79a2-7515-4de1-b7fa-f181ac2e9a14/volumes" Nov 02 21:59:56.028696 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.028636 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2"] Nov 02 21:59:56.029363 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029344 5091 cpu_manager.go:414] "containerMap: removing container" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" Nov 02 21:59:56.029363 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029364 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" Nov 02 21:59:56.029481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029378 5091 cpu_manager.go:414] "containerMap: removing container" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" Nov 02 21:59:56.029481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029383 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" Nov 02 21:59:56.029481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029396 5091 cpu_manager.go:414] "containerMap: removing container" podUID="01ac0adf-5e90-4795-8bfa-a6ff46d936a5" containerName="revoke-keys" Nov 02 21:59:56.029481 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029401 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ac0adf-5e90-4795-8bfa-a6ff46d936a5" containerName="revoke-keys" Nov 02 21:59:56.029605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029525 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" Nov 02 21:59:56.029605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029532 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="544748cb-be52-4ce3-8772-78998f8b5fe8" containerName="istio-proxy" Nov 02 21:59:56.029605 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.029539 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ac0adf-5e90-4795-8bfa-a6ff46d936a5" containerName="revoke-keys" Nov 02 21:59:56.034555 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.034537 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.154400 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.154361 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2"] Nov 02 21:59:56.205501 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205470 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.205700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205531 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.205700 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205658 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.205798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205721 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.205798 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205765 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.205884 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.205925 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.205918 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwjw\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.206020 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.206008 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.206066 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.206032 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.306992 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307033 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307083 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307080 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307122 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307186 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307222 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307380 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307332 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwjw\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307604 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307788 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307645 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307847 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307833 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.307908 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.307861 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.308094 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.308070 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.309909 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.309881 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.310047 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.310008 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.363969 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.363933 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.364120 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.364102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwjw\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw\") pod \"maas-default-gateway-openshift-default-65c448b547-dzsk2\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:56.649284 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:56.649250 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:57.100600 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:57.100523 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2"] Nov 02 21:59:57.138436 ip-10-0-128-36 kubenswrapper[5091]: W1102 21:59:57.138409 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a59afb9_6bdc_4761_89f6_931cee41d27b.slice/crio-2ff8bd7eb4bc9c59ef93fa763699798d420626459d6e217fe4d95d73457d6083 WatchSource:0}: Error finding container 2ff8bd7eb4bc9c59ef93fa763699798d420626459d6e217fe4d95d73457d6083: Status 404 returned error can't find the container with id 2ff8bd7eb4bc9c59ef93fa763699798d420626459d6e217fe4d95d73457d6083 Nov 02 21:59:57.140917 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:57.140885 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:57.141001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:57.140955 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:57.141001 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:57.140987 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:57.785699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:57.783904 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" event={"ID":"1a59afb9-6bdc-4761-89f6-931cee41d27b","Type":"ContainerStarted","Data":"9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5"} Nov 02 21:59:57.785699 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:57.784006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" event={"ID":"1a59afb9-6bdc-4761-89f6-931cee41d27b","Type":"ContainerStarted","Data":"2ff8bd7eb4bc9c59ef93fa763699798d420626459d6e217fe4d95d73457d6083"} Nov 02 21:59:58.003794 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:58.003742 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podStartSLOduration=3.003728927 podStartE2EDuration="3.003728927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:57.950613408 +0000 UTC m=+1545.112708528" watchObservedRunningTime="2026-11-02 21:59:58.003728927 +0000 UTC m=+1545.165824041" Nov 02 21:59:58.649751 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:58.649705 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 21:59:58.651213 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:58.651190 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 21:59:58.651291 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:58.651235 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 21:59:59.649819 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:59.649782 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 21:59:59.650211 ip-10-0-128-36 kubenswrapper[5091]: I1102 21:59:59.649844 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 22:00:00.202292 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.202258 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk"] Nov 02 22:00:00.208091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.208065 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.208712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.208665 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6"] Nov 02 22:00:00.212942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.212924 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.213474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.213455 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd"] Nov 02 22:00:00.218004 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.217985 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.248454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.248430 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r"] Nov 02 22:00:00.253864 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.253844 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.254531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.254516 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w"] Nov 02 22:00:00.258874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.258853 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.259244 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.259226 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d"] Nov 02 22:00:00.283691 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.263289 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.284038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284009 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2vs\" (UniqueName: \"kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.284253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284230 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xp8k\" (UniqueName: \"kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.284402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284386 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.284606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284590 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5qh\" (UniqueName: \"kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh\") pod \"maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.284749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284732 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.284861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284847 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzv9\" (UniqueName: \"kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.284988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.284973 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.289828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.289801 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.290065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.290029 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmx2g\" (UniqueName: \"kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.290176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.279989 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:00:00.290176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.279913 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.290176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.286822 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2"] Nov 02 22:00:00.293943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.293919 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:00:00.294041 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.293970 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:00:00.296752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.296735 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.324913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.324890 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd"] Nov 02 22:00:00.329843 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.329815 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6"] Nov 02 22:00:00.337070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.337045 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk"] Nov 02 22:00:00.399957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.399917 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcbt\" (UniqueName: \"kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt\") pod \"maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.402095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.399989 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.402428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.400958 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.403012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.402381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmx2g\" (UniqueName: \"kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.403443 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.403256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.403632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.403604 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.404475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.404455 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2vs\" (UniqueName: \"kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.404849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.404830 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.405072 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.405054 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp8k\" (UniqueName: \"kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.405575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.405541 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.406749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.406729 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6lp4\" (UniqueName: \"kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4\") pod \"maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.407104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.406546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.407343 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.407325 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5qh\" (UniqueName: \"kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh\") pod \"maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.407820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.407801 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.408142 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.408123 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzv9\" (UniqueName: \"kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.408823 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.408308 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.409208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.409189 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.409577 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.408979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.409746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.408613 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.416802 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.416777 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w"] Nov 02 22:00:00.421749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.421725 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r"] Nov 02 22:00:00.426309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.426288 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2"] Nov 02 22:00:00.459327 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.459267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmx2g\" (UniqueName: \"kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.460492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.460370 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2vs\" (UniqueName: \"kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.499738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.498704 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d"] Nov 02 22:00:00.501069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.501043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xp8k\" (UniqueName: \"kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.501212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.501185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5qh\" (UniqueName: \"kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh\") pod \"maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.503398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.503374 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzv9\" (UniqueName: \"kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.510811 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.510776 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.510945 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.510853 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6lp4\" (UniqueName: \"kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4\") pod \"maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.511051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.510984 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcbt\" (UniqueName: \"kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt\") pod \"maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.511139 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.511094 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.512138 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.512113 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.513009 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.512985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.529145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.529121 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:00.538423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.538397 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:00.545827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.545807 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:00.596505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.596468 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:00.599893 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.599866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6lp4\" (UniqueName: \"kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4\") pod \"maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.600634 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.600456 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcbt\" (UniqueName: \"kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt\") pod \"maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.608593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.608332 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:00.615302 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.615090 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:00.630728 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.625490 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:00.704874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.703103 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:00:00.704874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:00.703162 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 22:00:01.150560 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.150533 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk"] Nov 02 22:00:01.152011 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:01.151982 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b78f4c2_6f31_467b_953f_8ded9f4010f3.slice/crio-dfe7840130f2d01fa19f5fcdf97386ff30b737fadaea379cfe78e1ad48d7e768 WatchSource:0}: Error finding container dfe7840130f2d01fa19f5fcdf97386ff30b737fadaea379cfe78e1ad48d7e768: Status 404 returned error can't find the container with id dfe7840130f2d01fa19f5fcdf97386ff30b737fadaea379cfe78e1ad48d7e768 Nov 02 22:00:01.205494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.205464 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6"] Nov 02 22:00:01.211521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.211498 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd"] Nov 02 22:00:01.217958 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:01.217929 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a1b5e1_d1c6_412f_899e_ddfdad7e6276.slice/crio-765c654305f5c51b92da2bdc7f9660e5d3a1baa54b6b75ceb74a7896c100456b WatchSource:0}: Error finding container 765c654305f5c51b92da2bdc7f9660e5d3a1baa54b6b75ceb74a7896c100456b: Status 404 returned error can't find the container with id 765c654305f5c51b92da2bdc7f9660e5d3a1baa54b6b75ceb74a7896c100456b Nov 02 22:00:01.255112 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.255089 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2"] Nov 02 22:00:01.260006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.259982 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r"] Nov 02 22:00:01.353475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.353438 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w"] Nov 02 22:00:01.398095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.398068 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d"] Nov 02 22:00:01.650209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.650150 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:00:01.650209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.650194 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 22:00:01.874706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.869945 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerStarted","Data":"6b9d7e472c30c06a44059fef855c0388a3efe94e3a02548d90b4f309ef4aced4"} Nov 02 22:00:01.874706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.870018 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerStarted","Data":"73b15b82e2f429da170c4b3809fabdf390ceaded8f8b14d00d2110fb3b5fcfe5"} Nov 02 22:00:01.893594 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.893501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerStarted","Data":"77dc277446ef81cc38ce5acef7dccff1d75711e1d82db89e79e2fc6491569ec7"} Nov 02 22:00:01.899725 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.895700 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" event={"ID":"165f4f58-43a4-48ae-b3dd-aea9ec798224","Type":"ContainerStarted","Data":"b3277de4bb575efef27b2c2e5bfd1ff84572d1131d7018aca1b2dc89137f5810"} Nov 02 22:00:01.899725 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.897566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerStarted","Data":"5f13cb5c56169d96e7617470ff60e5076d83747f6d729d87060b45d9ec260c31"} Nov 02 22:00:01.899725 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.897764 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerStarted","Data":"765c654305f5c51b92da2bdc7f9660e5d3a1baa54b6b75ceb74a7896c100456b"} Nov 02 22:00:01.899725 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.899490 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerStarted","Data":"badbe0019c476013dc468d63b5336f082f0ddf0e3c7ce874e5f4a9b94380fcfd"} Nov 02 22:00:01.899725 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.899525 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerStarted","Data":"dfe7840130f2d01fa19f5fcdf97386ff30b737fadaea379cfe78e1ad48d7e768"} Nov 02 22:00:01.901308 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.901241 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" event={"ID":"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7","Type":"ContainerStarted","Data":"5181fa6f40479c78dd5720d61641b800dc15196cc95270d7274fed8e47deea20"} Nov 02 22:00:01.902321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.902298 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" event={"ID":"7631d89e-fa15-4465-85e2-9da949645938","Type":"ContainerStarted","Data":"5a8a6576f1b057b646c463973e01f3eb02aa3ec9f39147db4365d7915ddfa945"} Nov 02 22:00:01.995370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:01.995309 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" podStartSLOduration=1.995291367 podStartE2EDuration="1.995291367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:01.993017894 +0000 UTC m=+1549.155113016" watchObservedRunningTime="2026-11-02 22:00:01.995291367 +0000 UTC m=+1549.157386487" Nov 02 22:00:02.077878 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.077815 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2"] Nov 02 22:00:02.096759 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.096731 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d134b0-qrfx2"] Nov 02 22:00:02.112770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.112724 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" podStartSLOduration=2.112711285 podStartE2EDuration="2.112711285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.110800537 +0000 UTC m=+1549.272895657" watchObservedRunningTime="2026-11-02 22:00:02.112711285 +0000 UTC m=+1549.274806405" Nov 02 22:00:02.224237 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.224182 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" podStartSLOduration=2.22416707 podStartE2EDuration="2.22416707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:02.221332224 +0000 UTC m=+1549.383427346" watchObservedRunningTime="2026-11-02 22:00:02.22416707 +0000 UTC m=+1549.386262221" Nov 02 22:00:02.649720 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.649689 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:00:02.649887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.649740 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 22:00:02.913569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.913540 5091 generic.go:356] "Generic (PLEG): container finished" podID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerID="b5f6509480c458abbcafcf62687a4005df3e6d66a0165c00f58c430e33c6cc1f" exitCode=60 Nov 02 22:00:02.913998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.913627 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" event={"ID":"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7","Type":"ContainerDied","Data":"b5f6509480c458abbcafcf62687a4005df3e6d66a0165c00f58c430e33c6cc1f"} Nov 02 22:00:02.913998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.913957 5091 scope.go:122] "RemoveContainer" containerID="b5f6509480c458abbcafcf62687a4005df3e6d66a0165c00f58c430e33c6cc1f" Nov 02 22:00:02.915479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.915455 5091 generic.go:356] "Generic (PLEG): container finished" podID="7631d89e-fa15-4465-85e2-9da949645938" containerID="2cb2c9e1ede4f78fee821c4aaff8d7029bb4f5f01b0249c5c951760b7cdbfa48" exitCode=60 Nov 02 22:00:02.915583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.915529 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" event={"ID":"7631d89e-fa15-4465-85e2-9da949645938","Type":"ContainerDied","Data":"2cb2c9e1ede4f78fee821c4aaff8d7029bb4f5f01b0249c5c951760b7cdbfa48"} Nov 02 22:00:02.915849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.915830 5091 scope.go:122] "RemoveContainer" containerID="2cb2c9e1ede4f78fee821c4aaff8d7029bb4f5f01b0249c5c951760b7cdbfa48" Nov 02 22:00:02.917457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.917430 5091 generic.go:356] "Generic (PLEG): container finished" podID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerID="6b9d7e472c30c06a44059fef855c0388a3efe94e3a02548d90b4f309ef4aced4" exitCode=60 Nov 02 22:00:02.917621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.917472 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerDied","Data":"6b9d7e472c30c06a44059fef855c0388a3efe94e3a02548d90b4f309ef4aced4"} Nov 02 22:00:02.917935 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.917830 5091 scope.go:122] "RemoveContainer" containerID="6b9d7e472c30c06a44059fef855c0388a3efe94e3a02548d90b4f309ef4aced4" Nov 02 22:00:02.919438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.919360 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerStarted","Data":"5d9e7900320bbaacd44b3fc5335b74c9c24f5ee28d68bd311056f9feef6bd991"} Nov 02 22:00:02.921026 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.921008 5091 generic.go:356] "Generic (PLEG): container finished" podID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerID="c77454aceee39feed06cdd1aeb573ca927cf26256eca23ab3c98b5baf99a999e" exitCode=60 Nov 02 22:00:02.921130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.921091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" event={"ID":"165f4f58-43a4-48ae-b3dd-aea9ec798224","Type":"ContainerDied","Data":"c77454aceee39feed06cdd1aeb573ca927cf26256eca23ab3c98b5baf99a999e"} Nov 02 22:00:02.921322 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.921305 5091 scope.go:122] "RemoveContainer" containerID="c77454aceee39feed06cdd1aeb573ca927cf26256eca23ab3c98b5baf99a999e" Nov 02 22:00:02.922736 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.922716 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerID="5f13cb5c56169d96e7617470ff60e5076d83747f6d729d87060b45d9ec260c31" exitCode=60 Nov 02 22:00:02.922832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.922796 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerDied","Data":"5f13cb5c56169d96e7617470ff60e5076d83747f6d729d87060b45d9ec260c31"} Nov 02 22:00:02.922991 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.922977 5091 scope.go:122] "RemoveContainer" containerID="5f13cb5c56169d96e7617470ff60e5076d83747f6d729d87060b45d9ec260c31" Nov 02 22:00:02.924526 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.924505 5091 generic.go:356] "Generic (PLEG): container finished" podID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerID="badbe0019c476013dc468d63b5336f082f0ddf0e3c7ce874e5f4a9b94380fcfd" exitCode=60 Nov 02 22:00:02.924600 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.924550 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerDied","Data":"badbe0019c476013dc468d63b5336f082f0ddf0e3c7ce874e5f4a9b94380fcfd"} Nov 02 22:00:02.924791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:02.924778 5091 scope.go:122] "RemoveContainer" containerID="badbe0019c476013dc468d63b5336f082f0ddf0e3c7ce874e5f4a9b94380fcfd" Nov 02 22:00:03.019728 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.019615 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff220e65-774a-44e9-85e8-85514e205eaa" path="/var/lib/kubelet/pods/ff220e65-774a-44e9-85e8-85514e205eaa/volumes" Nov 02 22:00:03.222346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.222289 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" podStartSLOduration=3.222268289 podStartE2EDuration="3.222268289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.205365967 +0000 UTC m=+1550.367461093" watchObservedRunningTime="2026-11-02 22:00:03.222268289 +0000 UTC m=+1550.384363413" Nov 02 22:00:03.652234 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.652193 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:00:03.652439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.652270 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 22:00:03.940828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.940792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerStarted","Data":"23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9"} Nov 02 22:00:03.943167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.943142 5091 generic.go:356] "Generic (PLEG): container finished" podID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerID="5d9e7900320bbaacd44b3fc5335b74c9c24f5ee28d68bd311056f9feef6bd991" exitCode=60 Nov 02 22:00:03.943293 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.943230 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerDied","Data":"5d9e7900320bbaacd44b3fc5335b74c9c24f5ee28d68bd311056f9feef6bd991"} Nov 02 22:00:03.943517 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.943501 5091 scope.go:122] "RemoveContainer" containerID="5d9e7900320bbaacd44b3fc5335b74c9c24f5ee28d68bd311056f9feef6bd991" Nov 02 22:00:03.945737 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.945712 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" event={"ID":"165f4f58-43a4-48ae-b3dd-aea9ec798224","Type":"ContainerStarted","Data":"9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9"} Nov 02 22:00:03.949074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.949050 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerStarted","Data":"6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2"} Nov 02 22:00:03.951246 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.951224 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerStarted","Data":"ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a"} Nov 02 22:00:03.953860 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.953835 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" event={"ID":"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7","Type":"ContainerStarted","Data":"04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036"} Nov 02 22:00:03.956720 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:03.956655 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" event={"ID":"7631d89e-fa15-4465-85e2-9da949645938","Type":"ContainerStarted","Data":"ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1"} Nov 02 22:00:04.093340 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.093289 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a1b5e1_d1c6_412f_899e_ddfdad7e6276.slice/crio-6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:04.286040 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.285975 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" podStartSLOduration=4.285953986 podStartE2EDuration="4.285953986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.281082568 +0000 UTC m=+1551.443177690" watchObservedRunningTime="2026-11-02 22:00:04.285953986 +0000 UTC m=+1551.448049107" Nov 02 22:00:04.394910 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.394841 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" podStartSLOduration=4.394826074 podStartE2EDuration="4.394826074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.393573831 +0000 UTC m=+1551.555668946" watchObservedRunningTime="2026-11-02 22:00:04.394826074 +0000 UTC m=+1551.556921194" Nov 02 22:00:04.506144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.506096 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" podStartSLOduration=4.506081017 podStartE2EDuration="4.506081017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.504750774 +0000 UTC m=+1551.666845895" watchObservedRunningTime="2026-11-02 22:00:04.506081017 +0000 UTC m=+1551.668176136" Nov 02 22:00:04.650160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.650126 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:00:04.650295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.650175 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 22:00:04.971629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.971596 5091 generic.go:356] "Generic (PLEG): container finished" podID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerID="04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036" exitCode=60 Nov 02 22:00:04.972088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.971960 5091 scope.go:122] "RemoveContainer" containerID="04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036" Nov 02 22:00:04.972306 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.972281 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w_odh-ai-gateway-infra(28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" Nov 02 22:00:04.973938 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.973912 5091 generic.go:356] "Generic (PLEG): container finished" podID="7631d89e-fa15-4465-85e2-9da949645938" containerID="ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1" exitCode=60 Nov 02 22:00:04.974294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.974272 5091 scope.go:122] "RemoveContainer" containerID="ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1" Nov 02 22:00:04.974634 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.974603 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r_odh-ai-gateway-infra(7631d89e-fa15-4465-85e2-9da949645938)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" podUID="7631d89e-fa15-4465-85e2-9da949645938" Nov 02 22:00:04.976139 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.976118 5091 generic.go:356] "Generic (PLEG): container finished" podID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerID="23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9" exitCode=60 Nov 02 22:00:04.976447 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.976426 5091 scope.go:122] "RemoveContainer" containerID="23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9" Nov 02 22:00:04.976773 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.976748 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6_odh-ai-gateway-infra(115f6684-aa14-4003-86a1-5775a5bd28a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" Nov 02 22:00:04.980232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.980213 5091 generic.go:356] "Generic (PLEG): container finished" podID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerID="9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9" exitCode=60 Nov 02 22:00:04.980566 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.980550 5091 scope.go:122] "RemoveContainer" containerID="9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9" Nov 02 22:00:04.980829 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.980807 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2_odh-ai-gateway-infra(165f4f58-43a4-48ae-b3dd-aea9ec798224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" Nov 02 22:00:04.981369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.981347 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" event={"ID":"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7","Type":"ContainerDied","Data":"04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036"} Nov 02 22:00:04.981464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.981392 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" event={"ID":"7631d89e-fa15-4465-85e2-9da949645938","Type":"ContainerDied","Data":"ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1"} Nov 02 22:00:04.981464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.981415 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerDied","Data":"23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9"} Nov 02 22:00:04.981464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.981440 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerStarted","Data":"588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46"} Nov 02 22:00:04.981464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.981456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" event={"ID":"165f4f58-43a4-48ae-b3dd-aea9ec798224","Type":"ContainerDied","Data":"9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9"} Nov 02 22:00:04.981464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.981458 5091 scope.go:122] "RemoveContainer" containerID="b5f6509480c458abbcafcf62687a4005df3e6d66a0165c00f58c430e33c6cc1f" Nov 02 22:00:04.982748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.982389 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerID="6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2" exitCode=60 Nov 02 22:00:04.982748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.982469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerDied","Data":"6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2"} Nov 02 22:00:04.982748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.982636 5091 scope.go:122] "RemoveContainer" containerID="6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2" Nov 02 22:00:04.982913 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.982854 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd_odh-ai-gateway-infra(c1a1b5e1-d1c6-412f-899e-ddfdad7e6276)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" Nov 02 22:00:04.984625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.984604 5091 generic.go:356] "Generic (PLEG): container finished" podID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerID="ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a" exitCode=60 Nov 02 22:00:04.984759 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.984662 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerDied","Data":"ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a"} Nov 02 22:00:04.984950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.984933 5091 scope.go:122] "RemoveContainer" containerID="ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a" Nov 02 22:00:04.985231 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:04.985209 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk_odh-ai-gateway-infra(3b78f4c2-6f31-467b-953f-8ded9f4010f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" Nov 02 22:00:04.998281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:04.998261 5091 scope.go:122] "RemoveContainer" containerID="2cb2c9e1ede4f78fee821c4aaff8d7029bb4f5f01b0249c5c951760b7cdbfa48" Nov 02 22:00:05.014582 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.014561 5091 scope.go:122] "RemoveContainer" containerID="6b9d7e472c30c06a44059fef855c0388a3efe94e3a02548d90b4f309ef4aced4" Nov 02 22:00:05.031521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.031499 5091 scope.go:122] "RemoveContainer" containerID="c77454aceee39feed06cdd1aeb573ca927cf26256eca23ab3c98b5baf99a999e" Nov 02 22:00:05.045619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.045598 5091 scope.go:122] "RemoveContainer" containerID="5f13cb5c56169d96e7617470ff60e5076d83747f6d729d87060b45d9ec260c31" Nov 02 22:00:05.063761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.063728 5091 scope.go:122] "RemoveContainer" containerID="badbe0019c476013dc468d63b5336f082f0ddf0e3c7ce874e5f4a9b94380fcfd" Nov 02 22:00:05.958590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.958557 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 22:00:05.959135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.959112 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 22:00:05.960173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.960143 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 22:00:05.998229 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:05.998207 5091 scope.go:122] "RemoveContainer" containerID="23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9" Nov 02 22:00:05.998632 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:05.998574 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6_odh-ai-gateway-infra(115f6684-aa14-4003-86a1-5775a5bd28a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" Nov 02 22:00:06.000421 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.000387 5091 generic.go:356] "Generic (PLEG): container finished" podID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerID="588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46" exitCode=60 Nov 02 22:00:06.000565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.000482 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerDied","Data":"588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46"} Nov 02 22:00:06.000565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.000549 5091 scope.go:122] "RemoveContainer" containerID="5d9e7900320bbaacd44b3fc5335b74c9c24f5ee28d68bd311056f9feef6bd991" Nov 02 22:00:06.000781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.000763 5091 scope.go:122] "RemoveContainer" containerID="588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46" Nov 02 22:00:06.001084 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:06.001057 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d_odh-ai-gateway-infra(67047f01-5a11-4b1f-9024-bf959ce43db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" Nov 02 22:00:06.002806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.002790 5091 scope.go:122] "RemoveContainer" containerID="9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9" Nov 02 22:00:06.003026 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:06.003006 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2_odh-ai-gateway-infra(165f4f58-43a4-48ae-b3dd-aea9ec798224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" Nov 02 22:00:06.004294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.004278 5091 scope.go:122] "RemoveContainer" containerID="6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2" Nov 02 22:00:06.004531 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:06.004508 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd_odh-ai-gateway-infra(c1a1b5e1-d1c6-412f-899e-ddfdad7e6276)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" Nov 02 22:00:06.006074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.006056 5091 scope.go:122] "RemoveContainer" containerID="ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a" Nov 02 22:00:06.006310 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:06.006289 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk_odh-ai-gateway-infra(3b78f4c2-6f31-467b-953f-8ded9f4010f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" Nov 02 22:00:06.007834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.007814 5091 scope.go:122] "RemoveContainer" containerID="04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036" Nov 02 22:00:06.008085 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:06.008055 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w_odh-ai-gateway-infra(28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" Nov 02 22:00:06.009539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.009510 5091 scope.go:122] "RemoveContainer" containerID="ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1" Nov 02 22:00:06.009756 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:06.009737 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r_odh-ai-gateway-infra(7631d89e-fa15-4465-85e2-9da949645938)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" podUID="7631d89e-fa15-4465-85e2-9da949645938" Nov 02 22:00:06.048183 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.048149 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f"] Nov 02 22:00:06.053605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:06.053570 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09e756c-8fb5f"] Nov 02 22:00:07.035654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:07.035620 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d" path="/var/lib/kubelet/pods/5f5b7e6c-af5c-485c-9fe6-1fbebba89a9d/volumes" Nov 02 22:00:07.042118 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:07.042089 5091 scope.go:122] "RemoveContainer" containerID="588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46" Nov 02 22:00:07.042445 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:07.042423 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d_odh-ai-gateway-infra(67047f01-5a11-4b1f-9024-bf959ce43db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" Nov 02 22:00:07.117351 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:07.117313 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:00:09.046024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:09.045993 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz"] Nov 02 22:00:09.095233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:09.095202 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49c481d-4rmjz"] Nov 02 22:00:10.978972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:10.978941 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3af708-2288-4621-a7ed-535e1700ca97" path="/var/lib/kubelet/pods/6e3af708-2288-4621-a7ed-535e1700ca97/volumes" Nov 02 22:00:11.046160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:11.046125 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz"] Nov 02 22:00:11.095590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:11.095558 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0822b887-c7zsz"] Nov 02 22:00:12.980280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:12.980248 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730350df-3b31-44d5-8592-604f25e4917a" path="/var/lib/kubelet/pods/730350df-3b31-44d5-8592-604f25e4917a/volumes" Nov 02 22:00:14.969586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:14.969555 5091 scope.go:122] "RemoveContainer" containerID="ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a" Nov 02 22:00:14.970028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:14.969748 5091 scope.go:122] "RemoveContainer" containerID="6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2" Nov 02 22:00:16.143569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.143535 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerID="5ae33b018a67a6b966db926c21d70466ffd9ddc241e63a774240ea95964b07e6" exitCode=60 Nov 02 22:00:16.144069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.143610 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerDied","Data":"5ae33b018a67a6b966db926c21d70466ffd9ddc241e63a774240ea95964b07e6"} Nov 02 22:00:16.144069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.143695 5091 scope.go:122] "RemoveContainer" containerID="6aebb97e9300da402bedbd340c126858eabd0e1d57467d52a301f3b1d3109fc2" Nov 02 22:00:16.144192 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.144107 5091 scope.go:122] "RemoveContainer" containerID="5ae33b018a67a6b966db926c21d70466ffd9ddc241e63a774240ea95964b07e6" Nov 02 22:00:16.144496 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:16.144466 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd_odh-ai-gateway-infra(c1a1b5e1-d1c6-412f-899e-ddfdad7e6276)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" Nov 02 22:00:16.147151 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.147130 5091 generic.go:356] "Generic (PLEG): container finished" podID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerID="d3943c7a6e8c3497814118b3b8447080b7fcdfdf77514527846ff71fb8684823" exitCode=60 Nov 02 22:00:16.147275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.147206 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerDied","Data":"d3943c7a6e8c3497814118b3b8447080b7fcdfdf77514527846ff71fb8684823"} Nov 02 22:00:16.147574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.147555 5091 scope.go:122] "RemoveContainer" containerID="d3943c7a6e8c3497814118b3b8447080b7fcdfdf77514527846ff71fb8684823" Nov 02 22:00:16.147866 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:16.147843 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk_odh-ai-gateway-infra(3b78f4c2-6f31-467b-953f-8ded9f4010f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" Nov 02 22:00:16.161258 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.161236 5091 scope.go:122] "RemoveContainer" containerID="ddfdd2b32d485e9b1c8c22651941b4a040a351e8b8e215fc360e76d60c2d693a" Nov 02 22:00:16.984730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.984699 5091 scope.go:122] "RemoveContainer" containerID="588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46" Nov 02 22:00:16.984940 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.984922 5091 scope.go:122] "RemoveContainer" containerID="9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9" Nov 02 22:00:16.985071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:16.985045 5091 scope.go:122] "RemoveContainer" containerID="04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036" Nov 02 22:00:17.272264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.272183 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd"] Nov 02 22:00:17.388548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.387487 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk"] Nov 02 22:00:17.455117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.453833 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:17.550068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.549999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs\" (UniqueName: \"kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs\") pod \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " Nov 02 22:00:17.550213 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.550106 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca\") pod \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\" (UID: \"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276\") " Nov 02 22:00:17.550463 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.550438 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca" pod "c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" (UID: "c1a1b5e1-d1c6-412f-899e-ddfdad7e6276"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:17.550840 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.550821 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.568773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.564247 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs" pod "c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" (UID: "c1a1b5e1-d1c6-412f-899e-ddfdad7e6276"). InnerVolumeSpecName "kube-api-access-gk2vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:17.652374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.652340 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk2vs\" (UniqueName: \"kubernetes.io/projected/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276-kube-api-access-gk2vs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.755600 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.755568 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:17.854858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.854782 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca\") pod \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " Nov 02 22:00:17.855086 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.855072 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9\" (UniqueName: \"kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9\") pod \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\" (UID: \"3b78f4c2-6f31-467b-953f-8ded9f4010f3\") " Nov 02 22:00:17.855131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.855091 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca" pod "3b78f4c2-6f31-467b-953f-8ded9f4010f3" (UID: "3b78f4c2-6f31-467b-953f-8ded9f4010f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:17.855454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.855435 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b78f4c2-6f31-467b-953f-8ded9f4010f3-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.857359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.857335 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9" pod "3b78f4c2-6f31-467b-953f-8ded9f4010f3" (UID: "3b78f4c2-6f31-467b-953f-8ded9f4010f3"). InnerVolumeSpecName "kube-api-access-vgzv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:17.957642 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.957605 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgzv9\" (UniqueName: \"kubernetes.io/projected/3b78f4c2-6f31-467b-953f-8ded9f4010f3-kube-api-access-vgzv9\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.970286 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.970260 5091 scope.go:122] "RemoveContainer" containerID="23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9" Nov 02 22:00:17.970586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:17.970565 5091 scope.go:122] "RemoveContainer" containerID="ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1" Nov 02 22:00:18.047398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.047369 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j"] Nov 02 22:00:18.052270 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.052247 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bc7ad71-54j7j"] Nov 02 22:00:18.153028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.152998 5091 scope.go:122] "RemoveContainer" containerID="8d9795252907ab6c099582f9eb39ae53deed0375883d1e8cb54ed23871bf362b" Nov 02 22:00:18.164592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.164560 5091 scope.go:122] "RemoveContainer" containerID="715b474e3f64a7f3b0eab7f12bee7e4d26e8102ceb52e89063fd3e7bc9dbbabb" Nov 02 22:00:18.173384 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.173362 5091 generic.go:356] "Generic (PLEG): container finished" podID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerID="8b3120287c15c5b3dc35e87892d7eedb83e9f83afaad73f7f0a0b34867567da4" exitCode=60 Nov 02 22:00:18.173493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.173443 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" event={"ID":"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7","Type":"ContainerDied","Data":"8b3120287c15c5b3dc35e87892d7eedb83e9f83afaad73f7f0a0b34867567da4"} Nov 02 22:00:18.173554 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.173509 5091 scope.go:122] "RemoveContainer" containerID="04bafc589f99615848cc580fcdeb0accf023e0736b86e3eef178b85bd0f80036" Nov 02 22:00:18.173983 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.173951 5091 scope.go:122] "RemoveContainer" containerID="8b3120287c15c5b3dc35e87892d7eedb83e9f83afaad73f7f0a0b34867567da4" Nov 02 22:00:18.174352 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:18.174329 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w_odh-ai-gateway-infra(28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" Nov 02 22:00:18.176920 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.176897 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerStarted","Data":"28acefd9b280256eab310c426784f8e5a6a726ecea38cface5475494d5ed2002"} Nov 02 22:00:18.179258 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.179235 5091 generic.go:356] "Generic (PLEG): container finished" podID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerID="cb81c1d5e87488450db67769a327b38a373521b961868c6532cf87727c852765" exitCode=60 Nov 02 22:00:18.179366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.179313 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" event={"ID":"165f4f58-43a4-48ae-b3dd-aea9ec798224","Type":"ContainerDied","Data":"cb81c1d5e87488450db67769a327b38a373521b961868c6532cf87727c852765"} Nov 02 22:00:18.179740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.179722 5091 scope.go:122] "RemoveContainer" containerID="cb81c1d5e87488450db67769a327b38a373521b961868c6532cf87727c852765" Nov 02 22:00:18.180084 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:18.180062 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2_odh-ai-gateway-infra(165f4f58-43a4-48ae-b3dd-aea9ec798224)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" Nov 02 22:00:18.180925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.180911 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" Nov 02 22:00:18.181032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.180947 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd" event={"ID":"c1a1b5e1-d1c6-412f-899e-ddfdad7e6276","Type":"ContainerDied","Data":"765c654305f5c51b92da2bdc7f9660e5d3a1baa54b6b75ceb74a7896c100456b"} Nov 02 22:00:18.182510 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.182487 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" event={"ID":"3b78f4c2-6f31-467b-953f-8ded9f4010f3","Type":"ContainerDied","Data":"dfe7840130f2d01fa19f5fcdf97386ff30b737fadaea379cfe78e1ad48d7e768"} Nov 02 22:00:18.182610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.182498 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk" Nov 02 22:00:18.225021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.224645 5091 scope.go:122] "RemoveContainer" containerID="aff7d5e9adbdc0c20543ed6ca12513c014fac3b4ed9f86e6ce7730e619538704" Nov 02 22:00:18.225194 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.225082 5091 scope.go:122] "RemoveContainer" containerID="9243ef251cf606985d201a93b38d9dc4c80211de62795ea1bc5bca3e6755c2e9" Nov 02 22:00:18.241147 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.241124 5091 scope.go:122] "RemoveContainer" containerID="90f2262b0a230a416ff4eb7bdd4254169e2b45080f3cb7ab7549fdf9248457b2" Nov 02 22:00:18.241472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.241456 5091 scope.go:122] "RemoveContainer" containerID="5ae33b018a67a6b966db926c21d70466ffd9ddc241e63a774240ea95964b07e6" Nov 02 22:00:18.256110 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.256092 5091 scope.go:122] "RemoveContainer" containerID="d3943c7a6e8c3497814118b3b8447080b7fcdfdf77514527846ff71fb8684823" Nov 02 22:00:18.402827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.402798 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd"] Nov 02 22:00:18.411169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.411148 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894280-ppbnd"] Nov 02 22:00:18.744439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.744361 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk"] Nov 02 22:00:18.749218 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.749193 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894280-5qpwk"] Nov 02 22:00:18.978606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.978575 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" path="/var/lib/kubelet/pods/3b78f4c2-6f31-467b-953f-8ded9f4010f3/volumes" Nov 02 22:00:18.979136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.979119 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8cc440-57d5-4ca1-99c0-07a4b1f8262b" path="/var/lib/kubelet/pods/8b8cc440-57d5-4ca1-99c0-07a4b1f8262b/volumes" Nov 02 22:00:18.979517 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:18.979504 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" path="/var/lib/kubelet/pods/c1a1b5e1-d1c6-412f-899e-ddfdad7e6276/volumes" Nov 02 22:00:19.194140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.194101 5091 generic.go:356] "Generic (PLEG): container finished" podID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerID="376d583e23972f960274ad183ee3eaf8e6dce6eec3595896ced623acd90e37e9" exitCode=60 Nov 02 22:00:19.194309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.194159 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerDied","Data":"376d583e23972f960274ad183ee3eaf8e6dce6eec3595896ced623acd90e37e9"} Nov 02 22:00:19.194309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.194224 5091 scope.go:122] "RemoveContainer" containerID="23d93ada3df562652619cfd028559a5c69ce854e06874ebadf18b19672ea8bc9" Nov 02 22:00:19.194577 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.194562 5091 scope.go:122] "RemoveContainer" containerID="376d583e23972f960274ad183ee3eaf8e6dce6eec3595896ced623acd90e37e9" Nov 02 22:00:19.194892 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:19.194872 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6_odh-ai-gateway-infra(115f6684-aa14-4003-86a1-5775a5bd28a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" Nov 02 22:00:19.196610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.196589 5091 generic.go:356] "Generic (PLEG): container finished" podID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerID="28acefd9b280256eab310c426784f8e5a6a726ecea38cface5475494d5ed2002" exitCode=60 Nov 02 22:00:19.196769 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.196658 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerDied","Data":"28acefd9b280256eab310c426784f8e5a6a726ecea38cface5475494d5ed2002"} Nov 02 22:00:19.196993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.196979 5091 scope.go:122] "RemoveContainer" containerID="28acefd9b280256eab310c426784f8e5a6a726ecea38cface5475494d5ed2002" Nov 02 22:00:19.197283 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:19.197263 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d_odh-ai-gateway-infra(67047f01-5a11-4b1f-9024-bf959ce43db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" Nov 02 22:00:19.204190 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.204167 5091 generic.go:356] "Generic (PLEG): container finished" podID="7631d89e-fa15-4465-85e2-9da949645938" containerID="cabc49a06857fecde9bf945052af5e407a1f0dca724717698fb862fe528f216c" exitCode=60 Nov 02 22:00:19.204320 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.204226 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" event={"ID":"7631d89e-fa15-4465-85e2-9da949645938","Type":"ContainerDied","Data":"cabc49a06857fecde9bf945052af5e407a1f0dca724717698fb862fe528f216c"} Nov 02 22:00:19.204560 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.204543 5091 scope.go:122] "RemoveContainer" containerID="cabc49a06857fecde9bf945052af5e407a1f0dca724717698fb862fe528f216c" Nov 02 22:00:19.204830 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:19.204812 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r_odh-ai-gateway-infra(7631d89e-fa15-4465-85e2-9da949645938)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" podUID="7631d89e-fa15-4465-85e2-9da949645938" Nov 02 22:00:19.211042 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.211019 5091 scope.go:122] "RemoveContainer" containerID="588d51ccbcad4c00198caf5548a3517f09e956a31a2720749bf56300e071be46" Nov 02 22:00:19.227394 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.227367 5091 scope.go:122] "RemoveContainer" containerID="ec4648ccff4fee3865a644fbf97c099ef2421681010f6358e1ca81e42503f8e1" Nov 02 22:00:19.300894 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.300858 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w"] Nov 02 22:00:19.470615 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.470584 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:19.532962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.532932 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2"] Nov 02 22:00:19.581781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.581748 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k\" (UniqueName: \"kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k\") pod \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " Nov 02 22:00:19.582032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.582014 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca\") pod \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\" (UID: \"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7\") " Nov 02 22:00:19.582329 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.582303 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca" pod "28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" (UID: "28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:19.582683 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.582653 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.584000 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.583959 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k" pod "28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" (UID: "28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7"). InnerVolumeSpecName "kube-api-access-8xp8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:19.643513 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.643478 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d"] Nov 02 22:00:19.684623 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.684587 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xp8k\" (UniqueName: \"kubernetes.io/projected/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7-kube-api-access-8xp8k\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.707561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.707499 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:19.786187 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.786155 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4\" (UniqueName: \"kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4\") pod \"165f4f58-43a4-48ae-b3dd-aea9ec798224\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " Nov 02 22:00:19.786325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.786306 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca\") pod \"165f4f58-43a4-48ae-b3dd-aea9ec798224\" (UID: \"165f4f58-43a4-48ae-b3dd-aea9ec798224\") " Nov 02 22:00:19.786651 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.786630 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca" pod "165f4f58-43a4-48ae-b3dd-aea9ec798224" (UID: "165f4f58-43a4-48ae-b3dd-aea9ec798224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:19.786961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.786937 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165f4f58-43a4-48ae-b3dd-aea9ec798224-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.788287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.788261 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4" pod "165f4f58-43a4-48ae-b3dd-aea9ec798224" (UID: "165f4f58-43a4-48ae-b3dd-aea9ec798224"). InnerVolumeSpecName "kube-api-access-q6lp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:19.888628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:19.888595 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6lp4\" (UniqueName: \"kubernetes.io/projected/165f4f58-43a4-48ae-b3dd-aea9ec798224-kube-api-access-q6lp4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.214784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.214756 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" Nov 02 22:00:20.214950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.214789 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2" event={"ID":"165f4f58-43a4-48ae-b3dd-aea9ec798224","Type":"ContainerDied","Data":"b3277de4bb575efef27b2c2e5bfd1ff84572d1131d7018aca1b2dc89137f5810"} Nov 02 22:00:20.214950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.214855 5091 scope.go:122] "RemoveContainer" containerID="cb81c1d5e87488450db67769a327b38a373521b961868c6532cf87727c852765" Nov 02 22:00:20.216613 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.216592 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" Nov 02 22:00:20.217126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.216746 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w" event={"ID":"28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7","Type":"ContainerDied","Data":"5181fa6f40479c78dd5720d61641b800dc15196cc95270d7274fed8e47deea20"} Nov 02 22:00:20.229243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.229218 5091 scope.go:122] "RemoveContainer" containerID="8b3120287c15c5b3dc35e87892d7eedb83e9f83afaad73f7f0a0b34867567da4" Nov 02 22:00:20.321784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.321278 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6"] Nov 02 22:00:20.364747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.364721 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:20.495588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.495558 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2"] Nov 02 22:00:20.496944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.496921 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca\") pod \"67047f01-5a11-4b1f-9024-bf959ce43db0\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " Nov 02 22:00:20.497098 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.497083 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt\" (UniqueName: \"kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt\") pod \"67047f01-5a11-4b1f-9024-bf959ce43db0\" (UID: \"67047f01-5a11-4b1f-9024-bf959ce43db0\") " Nov 02 22:00:20.497260 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.497238 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca" pod "67047f01-5a11-4b1f-9024-bf959ce43db0" (UID: "67047f01-5a11-4b1f-9024-bf959ce43db0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.497635 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.497617 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f01-5a11-4b1f-9024-bf959ce43db0-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.499272 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.499217 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt" pod "67047f01-5a11-4b1f-9024-bf959ce43db0" (UID: "67047f01-5a11-4b1f-9024-bf959ce43db0"). InnerVolumeSpecName "kube-api-access-xkcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.500825 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.500803 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a4e90e53-29894280-k9xm2"] Nov 02 22:00:20.525693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.525660 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:20.543714 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.543690 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r"] Nov 02 22:00:20.599244 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.599213 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkcbt\" (UniqueName: \"kubernetes.io/projected/67047f01-5a11-4b1f-9024-bf959ce43db0-kube-api-access-xkcbt\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.610036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.609244 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w"] Nov 02 22:00:20.621399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.621369 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894280-k9c7w"] Nov 02 22:00:20.700762 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.700732 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh\" (UniqueName: \"kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh\") pod \"115f6684-aa14-4003-86a1-5775a5bd28a3\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " Nov 02 22:00:20.700884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.700854 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca\") pod \"115f6684-aa14-4003-86a1-5775a5bd28a3\" (UID: \"115f6684-aa14-4003-86a1-5775a5bd28a3\") " Nov 02 22:00:20.701171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.701146 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca" pod "115f6684-aa14-4003-86a1-5775a5bd28a3" (UID: "115f6684-aa14-4003-86a1-5775a5bd28a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.701523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.701502 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115f6684-aa14-4003-86a1-5775a5bd28a3-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.702901 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.702875 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh" pod "115f6684-aa14-4003-86a1-5775a5bd28a3" (UID: "115f6684-aa14-4003-86a1-5775a5bd28a3"). InnerVolumeSpecName "kube-api-access-6b5qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.717999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.717981 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:20.802857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.802761 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g\" (UniqueName: \"kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g\") pod \"7631d89e-fa15-4465-85e2-9da949645938\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " Nov 02 22:00:20.803052 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.803031 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca\") pod \"7631d89e-fa15-4465-85e2-9da949645938\" (UID: \"7631d89e-fa15-4465-85e2-9da949645938\") " Nov 02 22:00:20.803474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.803440 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca" pod "7631d89e-fa15-4465-85e2-9da949645938" (UID: "7631d89e-fa15-4465-85e2-9da949645938"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.803890 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.803867 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5qh\" (UniqueName: \"kubernetes.io/projected/115f6684-aa14-4003-86a1-5775a5bd28a3-kube-api-access-6b5qh\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.803992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.803897 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631d89e-fa15-4465-85e2-9da949645938-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.805541 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.805516 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g" pod "7631d89e-fa15-4465-85e2-9da949645938" (UID: "7631d89e-fa15-4465-85e2-9da949645938"). InnerVolumeSpecName "kube-api-access-mmx2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.905834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.905805 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmx2g\" (UniqueName: \"kubernetes.io/projected/7631d89e-fa15-4465-85e2-9da949645938-kube-api-access-mmx2g\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.990705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.987843 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" path="/var/lib/kubelet/pods/165f4f58-43a4-48ae-b3dd-aea9ec798224/volumes" Nov 02 22:00:20.990705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:20.988800 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" path="/var/lib/kubelet/pods/28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7/volumes" Nov 02 22:00:21.232658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.232621 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" event={"ID":"115f6684-aa14-4003-86a1-5775a5bd28a3","Type":"ContainerDied","Data":"73b15b82e2f429da170c4b3809fabdf390ceaded8f8b14d00d2110fb3b5fcfe5"} Nov 02 22:00:21.232658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.232642 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6" Nov 02 22:00:21.232922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.232744 5091 scope.go:122] "RemoveContainer" containerID="376d583e23972f960274ad183ee3eaf8e6dce6eec3595896ced623acd90e37e9" Nov 02 22:00:21.234498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.234467 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" Nov 02 22:00:21.234615 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.234491 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d" event={"ID":"67047f01-5a11-4b1f-9024-bf959ce43db0","Type":"ContainerDied","Data":"77dc277446ef81cc38ce5acef7dccff1d75711e1d82db89e79e2fc6491569ec7"} Nov 02 22:00:21.239085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.238624 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" Nov 02 22:00:21.239085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.238649 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r" event={"ID":"7631d89e-fa15-4465-85e2-9da949645938","Type":"ContainerDied","Data":"5a8a6576f1b057b646c463973e01f3eb02aa3ec9f39147db4365d7915ddfa945"} Nov 02 22:00:21.253606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.253586 5091 scope.go:122] "RemoveContainer" containerID="28acefd9b280256eab310c426784f8e5a6a726ecea38cface5475494d5ed2002" Nov 02 22:00:21.266109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.266082 5091 scope.go:122] "RemoveContainer" containerID="cabc49a06857fecde9bf945052af5e407a1f0dca724717698fb862fe528f216c" Nov 02 22:00:21.409692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.403170 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d"] Nov 02 22:00:21.413689 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.410093 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34089f5b-29894280-9hq8d"] Nov 02 22:00:21.476367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.476337 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2"] Nov 02 22:00:21.477650 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.477626 5091 cpu_manager.go:414] "containerMap: removing container" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.477827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.477811 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.477937 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.477926 5091 cpu_manager.go:414] "containerMap: removing container" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.478033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478011 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.478033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478034 5091 cpu_manager.go:414] "containerMap: removing container" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478042 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478058 5091 cpu_manager.go:414] "containerMap: removing container" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478065 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478076 5091 cpu_manager.go:414] "containerMap: removing container" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478083 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478097 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478104 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478114 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478123 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478139 5091 cpu_manager.go:414] "containerMap: removing container" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478145 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478155 5091 cpu_manager.go:414] "containerMap: removing container" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478162 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478171 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478185 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478230 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478237 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478254 5091 cpu_manager.go:414] "containerMap: removing container" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478261 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478277 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478284 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478322 5091 cpu_manager.go:414] "containerMap: removing container" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478330 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478343 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478349 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478359 5091 cpu_manager.go:414] "containerMap: removing container" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478366 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478386 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478393 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478417 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.478427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478424 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478601 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478613 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478623 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478632 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478642 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478650 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478660 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478683 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478693 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478704 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478710 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478718 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478725 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478739 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478748 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478760 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.479576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.478769 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.484796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.484776 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.495883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.495846 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh"] Nov 02 22:00:21.496726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496652 5091 cpu_manager.go:414] "containerMap: removing container" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.496726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496688 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.496726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496710 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.496726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496718 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b78f4c2-6f31-467b-953f-8ded9f4010f3" containerName="cleanup" Nov 02 22:00:21.496998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496746 5091 cpu_manager.go:414] "containerMap: removing container" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.496998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496754 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ccf4e1-c8ac-4fb5-8679-ac7d5cb93fb7" containerName="cleanup" Nov 02 22:00:21.496998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496918 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="165f4f58-43a4-48ae-b3dd-aea9ec798224" containerName="cleanup" Nov 02 22:00:21.496998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496934 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" containerName="cleanup" Nov 02 22:00:21.496998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496949 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a1b5e1-d1c6-412f-899e-ddfdad7e6276" containerName="cleanup" Nov 02 22:00:21.496998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.496965 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7631d89e-fa15-4465-85e2-9da949645938" containerName="cleanup" Nov 02 22:00:21.503432 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.502156 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.519109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.519082 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6"] Nov 02 22:00:21.525280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.524775 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f48c4d-29894280-f84c6"] Nov 02 22:00:21.539022 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.538998 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2"] Nov 02 22:00:21.571505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.571473 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh"] Nov 02 22:00:21.571857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.571832 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-34089f5b-openshift-default-dockercfg-hzzd7\"" Nov 02 22:00:21.619724 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.619689 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.619964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.619746 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbgs\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.619964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.619774 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.619964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.619825 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.619964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.619935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.619964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.619963 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620015 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620102 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620127 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620182 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620303 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620364 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620395 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620460 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p46\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.620592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.620529 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.621414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.621390 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg"] Nov 02 22:00:21.626640 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.626614 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.642995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.642276 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r"] Nov 02 22:00:21.648397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.648369 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894280-48r2r"] Nov 02 22:00:21.682240 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.682168 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg"] Nov 02 22:00:21.683608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.683578 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:00:21.683608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.683590 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:00:21.683860 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.683825 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:00:21.683976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.683959 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:00:21.721649 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721612 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.721857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721658 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.721857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgxx\" (UniqueName: \"kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.721857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721757 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.721857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721785 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.721857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721854 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.722113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.721897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722262 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722144 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722262 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722262 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722204 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722262 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722236 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722297 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722307 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722311 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722356 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722527 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722413 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722527 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722448 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722527 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722452 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p46\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722527 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722490 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722587 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbgs\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722864 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722840 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.722996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.722996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.722971 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.723109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.723055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.723109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.723065 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.723203 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.723185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.724522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.724494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.725062 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.725043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.725303 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.725284 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.725397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.725378 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.779185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.779099 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.779391 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.779358 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbgs\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.780156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.780131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.780378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.780349 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p46\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.811420 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.811394 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:21.824976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.824944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.825284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.825249 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.826523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.825352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgxx\" (UniqueName: \"kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.826523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.825799 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.826808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.826785 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:21.829328 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.829308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.882046 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.881983 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgxx\" (UniqueName: \"kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx\") pod \"maas-api-revoke-sub-a3b1c8da-gtpgg\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:21.949592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:21.949284 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:22.272401 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:22.272373 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec43bde_b150_4c1a_9b9c_19b241c69ede.slice/crio-9a3d985294a6e94c650865c944fa6e8f431ec4bd9486102cec39d5b1b2b3f467 WatchSource:0}: Error finding container 9a3d985294a6e94c650865c944fa6e8f431ec4bd9486102cec39d5b1b2b3f467: Status 404 returned error can't find the container with id 9a3d985294a6e94c650865c944fa6e8f431ec4bd9486102cec39d5b1b2b3f467 Nov 02 22:00:22.273241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.273211 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2"] Nov 02 22:00:22.276052 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.276010 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:22.276154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.276095 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:22.276154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.276138 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:22.280271 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.280246 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh"] Nov 02 22:00:22.280733 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:22.280702 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c11011_4005_4f80_94c3_454bc7f5de68.slice/crio-cd235a6252c6d65623a5ab8909a37937514d2846900865378f13ba1fe3bfc05d WatchSource:0}: Error finding container cd235a6252c6d65623a5ab8909a37937514d2846900865378f13ba1fe3bfc05d: Status 404 returned error can't find the container with id cd235a6252c6d65623a5ab8909a37937514d2846900865378f13ba1fe3bfc05d Nov 02 22:00:22.288448 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.288415 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:22.288543 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.288478 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:22.288543 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.288506 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:22.451578 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:22.451039 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9bfd17_0af7_46d7_8aa5_8ee9c897d02f.slice/crio-21f159e9539d129a4411949945e58dc454fabf796863c77b078f3a7b83f23883 WatchSource:0}: Error finding container 21f159e9539d129a4411949945e58dc454fabf796863c77b078f3a7b83f23883: Status 404 returned error can't find the container with id 21f159e9539d129a4411949945e58dc454fabf796863c77b078f3a7b83f23883 Nov 02 22:00:22.459355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.459327 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg"] Nov 02 22:00:22.619064 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.619028 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm"] Nov 02 22:00:22.626411 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.626382 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.679699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.679337 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm"] Nov 02 22:00:22.756495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.756455 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtnd\" (UniqueName: \"kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.756746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.756632 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.756964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.756936 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.858897 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.858321 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.859375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.859349 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.859597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.859578 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtnd\" (UniqueName: \"kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.860917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.860894 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.861833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.861807 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.916794 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.916715 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtnd\" (UniqueName: \"kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd\") pod \"maas-api-revoke-keys-e2e-ait-34089f5b-psndm\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:22.945764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:22.945731 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:23.001529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.001476 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115f6684-aa14-4003-86a1-5775a5bd28a3" path="/var/lib/kubelet/pods/115f6684-aa14-4003-86a1-5775a5bd28a3/volumes" Nov 02 22:00:23.002603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.002568 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67047f01-5a11-4b1f-9024-bf959ce43db0" path="/var/lib/kubelet/pods/67047f01-5a11-4b1f-9024-bf959ce43db0/volumes" Nov 02 22:00:23.003396 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.003375 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7631d89e-fa15-4465-85e2-9da949645938" path="/var/lib/kubelet/pods/7631d89e-fa15-4465-85e2-9da949645938/volumes" Nov 02 22:00:23.299959 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.299863 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" event={"ID":"84c11011-4005-4f80-94c3-454bc7f5de68","Type":"ContainerStarted","Data":"8bee2bed385041f76e11f9cec6bd703c4dafd6b1a22520f3e4403026c21b62f3"} Nov 02 22:00:23.301097 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.301068 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" event={"ID":"84c11011-4005-4f80-94c3-454bc7f5de68","Type":"ContainerStarted","Data":"cd235a6252c6d65623a5ab8909a37937514d2846900865378f13ba1fe3bfc05d"} Nov 02 22:00:23.306846 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.306817 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" event={"ID":"5ec43bde-b150-4c1a-9b9c-19b241c69ede","Type":"ContainerStarted","Data":"797fd3bfcd7c837d01593b8aa1c3563305d41810f374eee10e667af36ffa42a3"} Nov 02 22:00:23.307051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.307035 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" event={"ID":"5ec43bde-b150-4c1a-9b9c-19b241c69ede","Type":"ContainerStarted","Data":"9a3d985294a6e94c650865c944fa6e8f431ec4bd9486102cec39d5b1b2b3f467"} Nov 02 22:00:23.326435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.319226 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" event={"ID":"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f","Type":"ContainerStarted","Data":"8a35beb5a090b7206e14ed9c79a30064fdb4e76bc27d98da75e5be825ca55cf3"} Nov 02 22:00:23.326435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.319323 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" event={"ID":"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f","Type":"ContainerStarted","Data":"21f159e9539d129a4411949945e58dc454fabf796863c77b078f3a7b83f23883"} Nov 02 22:00:23.425746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.421993 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podStartSLOduration=2.421974101 podStartE2EDuration="2.421974101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:23.421082542 +0000 UTC m=+1570.583177691" watchObservedRunningTime="2026-11-02 22:00:23.421974101 +0000 UTC m=+1570.584069226" Nov 02 22:00:23.542413 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.542355 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podStartSLOduration=2.542340669 podStartE2EDuration="2.542340669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:23.54042819 +0000 UTC m=+1570.702523320" watchObservedRunningTime="2026-11-02 22:00:23.542340669 +0000 UTC m=+1570.704435860" Nov 02 22:00:23.618713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.618608 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm"] Nov 02 22:00:23.621704 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:23.621230 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0cdbde_914f_4d81_8dba_8c70f049fa5a.slice/crio-746393dac181fbd12346b2458639fa1ceb03443b42b52dd9484bc668c3dde063 WatchSource:0}: Error finding container 746393dac181fbd12346b2458639fa1ceb03443b42b52dd9484bc668c3dde063: Status 404 returned error can't find the container with id 746393dac181fbd12346b2458639fa1ceb03443b42b52dd9484bc668c3dde063 Nov 02 22:00:23.707699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.693365 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" podStartSLOduration=2.6933467159999998 podStartE2EDuration="2.693346716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:23.692887532 +0000 UTC m=+1570.854982664" watchObservedRunningTime="2026-11-02 22:00:23.693346716 +0000 UTC m=+1570.855441837" Nov 02 22:00:23.814022 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.813931 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:23.815218 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.815188 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:00:23.815448 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.815411 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 22:00:23.828029 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.827999 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:23.829709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.829686 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:00:23.829883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:23.829858 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 22:00:24.330061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.330024 5091 generic.go:356] "Generic (PLEG): container finished" podID="ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" containerID="8a35beb5a090b7206e14ed9c79a30064fdb4e76bc27d98da75e5be825ca55cf3" exitCode=0 Nov 02 22:00:24.330418 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.330397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" event={"ID":"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f","Type":"ContainerDied","Data":"8a35beb5a090b7206e14ed9c79a30064fdb4e76bc27d98da75e5be825ca55cf3"} Nov 02 22:00:24.332505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.332478 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" event={"ID":"5e0cdbde-914f-4d81-8dba-8c70f049fa5a","Type":"ContainerStarted","Data":"c832c9d30fda95fe29aa2abfb1fb6f45bfd35f2bd612e318d1fe29f8162b5f9a"} Nov 02 22:00:24.332628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.332531 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" event={"ID":"5e0cdbde-914f-4d81-8dba-8c70f049fa5a","Type":"ContainerStarted","Data":"746393dac181fbd12346b2458639fa1ceb03443b42b52dd9484bc668c3dde063"} Nov 02 22:00:24.814738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.814701 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:00:24.815356 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.815321 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 22:00:24.828180 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.828161 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:00:24.828322 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:24.828306 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 22:00:25.342422 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.342387 5091 generic.go:356] "Generic (PLEG): container finished" podID="5e0cdbde-914f-4d81-8dba-8c70f049fa5a" containerID="c832c9d30fda95fe29aa2abfb1fb6f45bfd35f2bd612e318d1fe29f8162b5f9a" exitCode=0 Nov 02 22:00:25.342847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.342816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" event={"ID":"5e0cdbde-914f-4d81-8dba-8c70f049fa5a","Type":"ContainerDied","Data":"c832c9d30fda95fe29aa2abfb1fb6f45bfd35f2bd612e318d1fe29f8162b5f9a"} Nov 02 22:00:25.654898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.654870 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:25.734491 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.734458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca\") pod \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " Nov 02 22:00:25.734701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.734569 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca\") pod \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " Nov 02 22:00:25.734770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.734718 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx\" (UniqueName: \"kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx\") pod \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\" (UID: \"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f\") " Nov 02 22:00:25.735838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.735812 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca" pod "ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" (UID: "ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:25.737355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.737333 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca" pod "ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" (UID: "ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:25.737559 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.737537 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx" pod "ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" (UID: "ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f"). InnerVolumeSpecName "kube-api-access-pzgxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:25.823694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.815391 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:00:25.823694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.815461 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 22:00:25.828828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.828799 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:00:25.828953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.828852 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 22:00:25.837384 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.837350 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:25.837531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.837388 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzgxx\" (UniqueName: \"kubernetes.io/projected/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-kube-api-access-pzgxx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:25.837531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:25.837406 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:26.352576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.352550 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" Nov 02 22:00:26.353212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.352733 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg" event={"ID":"ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f","Type":"ContainerDied","Data":"21f159e9539d129a4411949945e58dc454fabf796863c77b078f3a7b83f23883"} Nov 02 22:00:26.353212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.352982 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f159e9539d129a4411949945e58dc454fabf796863c77b078f3a7b83f23883" Nov 02 22:00:26.658708 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.658684 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:26.751550 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.751518 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd\" (UniqueName: \"kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd\") pod \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " Nov 02 22:00:26.751796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.751566 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca\") pod \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " Nov 02 22:00:26.751796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.751693 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca\") pod \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\" (UID: \"5e0cdbde-914f-4d81-8dba-8c70f049fa5a\") " Nov 02 22:00:26.752556 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.752525 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca" pod "5e0cdbde-914f-4d81-8dba-8c70f049fa5a" (UID: "5e0cdbde-914f-4d81-8dba-8c70f049fa5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:26.754477 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.754452 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd" pod "5e0cdbde-914f-4d81-8dba-8c70f049fa5a" (UID: "5e0cdbde-914f-4d81-8dba-8c70f049fa5a"). InnerVolumeSpecName "kube-api-access-cdtnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:26.755293 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.755271 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca" pod "5e0cdbde-914f-4d81-8dba-8c70f049fa5a" (UID: "5e0cdbde-914f-4d81-8dba-8c70f049fa5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:26.828299 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.828265 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:00:26.829324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.829284 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 22:00:26.829824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.829217 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:00:26.829988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.829973 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 22:00:26.853466 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.853442 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:26.853592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.853580 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:26.853658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:26.853650 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdtnd\" (UniqueName: \"kubernetes.io/projected/5e0cdbde-914f-4d81-8dba-8c70f049fa5a-kube-api-access-cdtnd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:27.050341 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.050307 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x"] Nov 02 22:00:27.054955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.054933 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a011a95a-mkk6x"] Nov 02 22:00:27.392761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.392710 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" event={"ID":"5e0cdbde-914f-4d81-8dba-8c70f049fa5a","Type":"ContainerDied","Data":"746393dac181fbd12346b2458639fa1ceb03443b42b52dd9484bc668c3dde063"} Nov 02 22:00:27.393048 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.393020 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746393dac181fbd12346b2458639fa1ceb03443b42b52dd9484bc668c3dde063" Nov 02 22:00:27.393266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.393250 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm" Nov 02 22:00:27.820181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.820092 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:00:27.820181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.820166 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 22:00:27.828755 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.828421 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:00:27.828755 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:27.828496 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 22:00:28.811843 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:28.811805 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:00:28.812006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:28.811865 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 22:00:28.828175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:28.828140 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:00:28.828334 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:28.828190 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 22:00:28.980055 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:28.980024 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9c0a47-af7e-4075-a0d5-9ae6767e1d23" path="/var/lib/kubelet/pods/1c9c0a47-af7e-4075-a0d5-9ae6767e1d23/volumes" Nov 02 22:00:29.817482 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:29.817449 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:29.817957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:29.817934 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:29.818648 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:29.818625 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:00:29.833258 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:29.833236 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:29.833495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:29.833476 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:29.834118 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:29.834101 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:00:30.045512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:30.045480 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm"] Nov 02 22:00:30.050150 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:30.050128 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f060ba6-bz9tm"] Nov 02 22:00:30.979094 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:30.979060 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2ce9ba-50a2-4961-8ef8-98963de14f56" path="/var/lib/kubelet/pods/cc2ce9ba-50a2-4961-8ef8-98963de14f56/volumes" Nov 02 22:00:33.050156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:33.050125 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp"] Nov 02 22:00:33.099818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:33.099783 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11ffbac-cf2zp"] Nov 02 22:00:34.638897 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.638860 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx"] Nov 02 22:00:34.639555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.639541 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" containerName="revoke-keys" Nov 02 22:00:34.639601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.639555 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" containerName="revoke-keys" Nov 02 22:00:34.639601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.639566 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5e0cdbde-914f-4d81-8dba-8c70f049fa5a" containerName="revoke-keys" Nov 02 22:00:34.639601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.639572 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0cdbde-914f-4d81-8dba-8c70f049fa5a" containerName="revoke-keys" Nov 02 22:00:34.639727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.639703 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" containerName="revoke-keys" Nov 02 22:00:34.639727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.639715 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0cdbde-914f-4d81-8dba-8c70f049fa5a" containerName="revoke-keys" Nov 02 22:00:34.643353 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.643333 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.701253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.701224 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx"] Nov 02 22:00:34.731497 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.731465 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.731623 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.731577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8kc\" (UniqueName: \"kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.731776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.731756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.743203 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.743180 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:00:34.745800 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.745774 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:00:34.745800 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.745797 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:00:34.745937 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.745834 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:00:34.832828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.832804 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.832969 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.832841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8kc\" (UniqueName: \"kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.832969 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.832917 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.833622 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.833597 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.835263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.835242 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.889079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.889022 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8kc\" (UniqueName: \"kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc\") pod \"maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.957407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.957376 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:34.981471 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:34.981444 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d26da1a-c280-409d-b4e1-25f034d00afa" path="/var/lib/kubelet/pods/0d26da1a-c280-409d-b4e1-25f034d00afa/volumes" Nov 02 22:00:35.358047 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:00:35.358020 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa03aac7_7678_490a_9d7b_dec6922e9125.slice/crio-35d58838bdad9d50fe3700f96c7f115a3b0d81cff053ab37dbcbc94bb5081e2d WatchSource:0}: Error finding container 35d58838bdad9d50fe3700f96c7f115a3b0d81cff053ab37dbcbc94bb5081e2d: Status 404 returned error can't find the container with id 35d58838bdad9d50fe3700f96c7f115a3b0d81cff053ab37dbcbc94bb5081e2d Nov 02 22:00:35.358719 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:35.358694 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx"] Nov 02 22:00:35.487983 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:35.487951 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" event={"ID":"aa03aac7-7678-490a-9d7b-dec6922e9125","Type":"ContainerStarted","Data":"35d58838bdad9d50fe3700f96c7f115a3b0d81cff053ab37dbcbc94bb5081e2d"} Nov 02 22:00:36.497796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:36.497759 5091 generic.go:356] "Generic (PLEG): container finished" podID="aa03aac7-7678-490a-9d7b-dec6922e9125" containerID="45d7deed6a62634da1b42d8f35d9dcb9ca253fdc5f241bdcf978576e911aa417" exitCode=0 Nov 02 22:00:36.498185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:36.497852 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" event={"ID":"aa03aac7-7678-490a-9d7b-dec6922e9125","Type":"ContainerDied","Data":"45d7deed6a62634da1b42d8f35d9dcb9ca253fdc5f241bdcf978576e911aa417"} Nov 02 22:00:37.055264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.055216 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr"] Nov 02 22:00:37.058220 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.058187 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ace6d-k5wrr"] Nov 02 22:00:37.114364 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:00:37.114328 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:00:37.739175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.739145 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:37.784175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.784101 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca\") pod \"aa03aac7-7678-490a-9d7b-dec6922e9125\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " Nov 02 22:00:37.784312 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.784217 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca\") pod \"aa03aac7-7678-490a-9d7b-dec6922e9125\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " Nov 02 22:00:37.784312 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.784264 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc\" (UniqueName: \"kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc\") pod \"aa03aac7-7678-490a-9d7b-dec6922e9125\" (UID: \"aa03aac7-7678-490a-9d7b-dec6922e9125\") " Nov 02 22:00:37.784581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.784561 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca" pod "aa03aac7-7678-490a-9d7b-dec6922e9125" (UID: "aa03aac7-7678-490a-9d7b-dec6922e9125"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:37.784989 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.784966 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:37.786202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.786172 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca" pod "aa03aac7-7678-490a-9d7b-dec6922e9125" (UID: "aa03aac7-7678-490a-9d7b-dec6922e9125"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:37.786402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.786379 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc" pod "aa03aac7-7678-490a-9d7b-dec6922e9125" (UID: "aa03aac7-7678-490a-9d7b-dec6922e9125"). InnerVolumeSpecName "kube-api-access-6h8kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:37.888154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.888079 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h8kc\" (UniqueName: \"kubernetes.io/projected/aa03aac7-7678-490a-9d7b-dec6922e9125-kube-api-access-6h8kc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:37.888154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:37.888120 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03aac7-7678-490a-9d7b-dec6922e9125-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:38.526080 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:38.526043 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" Nov 02 22:00:38.526274 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:38.526041 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx" event={"ID":"aa03aac7-7678-490a-9d7b-dec6922e9125","Type":"ContainerDied","Data":"35d58838bdad9d50fe3700f96c7f115a3b0d81cff053ab37dbcbc94bb5081e2d"} Nov 02 22:00:38.526274 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:38.526158 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d58838bdad9d50fe3700f96c7f115a3b0d81cff053ab37dbcbc94bb5081e2d" Nov 02 22:00:38.989537 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:38.989500 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8b135f-96a4-48d2-8bb8-6a916863619c" path="/var/lib/kubelet/pods/cb8b135f-96a4-48d2-8bb8-6a916863619c/volumes" Nov 02 22:00:42.050458 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:42.050413 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp"] Nov 02 22:00:42.095018 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:42.094981 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df420285-fzmwp"] Nov 02 22:00:42.980493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:42.980459 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2217ebb-c519-49ad-8f26-53e55ec54d8a" path="/var/lib/kubelet/pods/f2217ebb-c519-49ad-8f26-53e55ec54d8a/volumes" Nov 02 22:00:54.046876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:54.046840 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl"] Nov 02 22:00:54.095606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:54.095575 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12cf4174-mshtl"] Nov 02 22:00:54.978664 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:54.978632 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bd8392-45fe-4c9b-beb8-e80861302be6" path="/var/lib/kubelet/pods/f1bd8392-45fe-4c9b-beb8-e80861302be6/volumes" Nov 02 22:00:55.045146 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:55.045115 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d"] Nov 02 22:00:55.095033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:55.095002 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bfadc4-9295d"] Nov 02 22:00:56.048667 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:56.048630 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848"] Nov 02 22:00:56.097908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:56.097875 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07b1a83-pb848"] Nov 02 22:00:56.978636 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:56.978607 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f" path="/var/lib/kubelet/pods/8f38015d-ed9e-42fd-b4c1-4c6cbebcc26f/volumes" Nov 02 22:00:56.979084 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:00:56.979069 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf6f582-d150-497a-bdcb-f2a4a528786d" path="/var/lib/kubelet/pods/fbf6f582-d150-497a-bdcb-f2a4a528786d/volumes" Nov 02 22:01:01.045303 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:01.045271 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm"] Nov 02 22:01:01.094906 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:01.094873 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf00d86-bgrrm"] Nov 02 22:01:02.978416 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:02.978383 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf24783e-99da-4187-a9bc-902adb087a9b" path="/var/lib/kubelet/pods/bf24783e-99da-4187-a9bc-902adb087a9b/volumes" Nov 02 22:01:03.045479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:03.045449 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x"] Nov 02 22:01:03.094789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:03.094763 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325cdd9-xg85x"] Nov 02 22:01:04.979097 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:04.979057 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cc358c-2ab2-4429-a67e-7e13adce3d60" path="/var/lib/kubelet/pods/59cc358c-2ab2-4429-a67e-7e13adce3d60/volumes" Nov 02 22:01:07.046242 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:07.046205 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8"] Nov 02 22:01:07.095090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:07.095057 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d16649-c2jb8"] Nov 02 22:01:07.116019 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:01:07.115979 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:01:08.049721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:08.049655 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v"] Nov 02 22:01:08.099087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:08.099055 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5084719-8lk5v"] Nov 02 22:01:08.978178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:08.978147 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc9149b-5525-402f-b190-686b259d0975" path="/var/lib/kubelet/pods/5cc9149b-5525-402f-b190-686b259d0975/volumes" Nov 02 22:01:08.978628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:08.978614 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963a778f-929d-4353-9087-75f7e9cb1058" path="/var/lib/kubelet/pods/963a778f-929d-4353-9087-75f7e9cb1058/volumes" Nov 02 22:01:12.049339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:12.049303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc"] Nov 02 22:01:12.097267 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:12.097232 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd9eb87-2k7kc"] Nov 02 22:01:12.985776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:12.985738 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df4cf88-7749-4d55-b258-b6499342e7cf" path="/var/lib/kubelet/pods/5df4cf88-7749-4d55-b258-b6499342e7cf/volumes" Nov 02 22:01:13.300340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.300247 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm"] Nov 02 22:01:13.349188 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.349155 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34089f5b-psndm"] Nov 02 22:01:13.485624 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.485591 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk"] Nov 02 22:01:13.486833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.486804 5091 cpu_manager.go:414] "containerMap: removing container" podUID="aa03aac7-7678-490a-9d7b-dec6922e9125" containerName="revoke-keys" Nov 02 22:01:13.486833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.486831 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa03aac7-7678-490a-9d7b-dec6922e9125" containerName="revoke-keys" Nov 02 22:01:13.486998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.486973 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa03aac7-7678-490a-9d7b-dec6922e9125" containerName="revoke-keys" Nov 02 22:01:13.492309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.492287 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.543355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.543318 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk"] Nov 02 22:01:13.544162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.544028 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:01:13.544162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.544063 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:01:13.544162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.544103 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:01:13.544162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.544143 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:01:13.622351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.622323 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.622575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.622438 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l2b\" (UniqueName: \"kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.622575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.622525 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.724156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.724126 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l2b\" (UniqueName: \"kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.724331 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.724199 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.724375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.724361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.724864 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.724832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.726632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.726612 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.780717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.780653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l2b\" (UniqueName: \"kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b\") pod \"maas-api-revoke-sub-8bb83ef0-ql2wk\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.809768 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.809736 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:13.845598 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.844736 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh"] Nov 02 22:01:13.845598 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.845000 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" containerID="cri-o://8bee2bed385041f76e11f9cec6bd703c4dafd6b1a22520f3e4403026c21b62f3" gracePeriod=30 Nov 02 22:01:13.854152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.854122 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2"] Nov 02 22:01:13.854369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:13.854345 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" containerID="cri-o://797fd3bfcd7c837d01593b8aa1c3563305d41810f374eee10e667af36ffa42a3" gracePeriod=30 Nov 02 22:01:14.104307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.089725 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq"] Nov 02 22:01:14.104307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.096072 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec0d1a-qgwqq"] Nov 02 22:01:14.240144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.240112 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk"] Nov 02 22:01:14.240642 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:01:14.240616 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e51ec8_5426_4e79_a38f_c0ce22e3180b.slice/crio-4598ec2c74f25ed632f9710bf7a23a60a89f9b7ceab464c4526f8cca9932c2e9 WatchSource:0}: Error finding container 4598ec2c74f25ed632f9710bf7a23a60a89f9b7ceab464c4526f8cca9932c2e9: Status 404 returned error can't find the container with id 4598ec2c74f25ed632f9710bf7a23a60a89f9b7ceab464c4526f8cca9932c2e9 Nov 02 22:01:14.820131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.820098 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:01:14.820523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.820148 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:14.835436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.835365 5091 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 02 22:01:14.835436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.835412 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:14.979618 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.979586 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0cdbde-914f-4d81-8dba-8c70f049fa5a" path="/var/lib/kubelet/pods/5e0cdbde-914f-4d81-8dba-8c70f049fa5a/volumes" Nov 02 22:01:14.980205 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:14.980188 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af15d935-9a83-4729-9a65-461e62183464" path="/var/lib/kubelet/pods/af15d935-9a83-4729-9a65-461e62183464/volumes" Nov 02 22:01:15.001840 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.001804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" event={"ID":"04e51ec8-5426-4e79-a38f-c0ce22e3180b","Type":"ContainerStarted","Data":"8c6f726d6997757beaf77233e5c9ac7131e8daecb9083904dac90104c8ae4a3c"} Nov 02 22:01:15.002013 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.001871 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" event={"ID":"04e51ec8-5426-4e79-a38f-c0ce22e3180b","Type":"ContainerStarted","Data":"4598ec2c74f25ed632f9710bf7a23a60a89f9b7ceab464c4526f8cca9932c2e9"} Nov 02 22:01:15.122776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.122707 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" podStartSLOduration=2.122687513 podStartE2EDuration="2.122687513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:15.116157528 +0000 UTC m=+1622.278252648" watchObservedRunningTime="2026-11-02 22:01:15.122687513 +0000 UTC m=+1622.284782631" Nov 02 22:01:15.459774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.459732 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj"] Nov 02 22:01:15.464724 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.464701 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.524128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.523809 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj"] Nov 02 22:01:15.530032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.529993 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530041 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530154 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f96c\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530190 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530264 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530363 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.530504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.530482 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.569278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.569213 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-fb0367-openshift-default-dockercfg-2fpk5\"" Nov 02 22:01:15.631834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.631796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.631834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.631833 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.631898 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.631928 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f96c\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.631959 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.631976 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.632008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.632073 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632440 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.632152 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632440 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.632281 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.632563 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.632857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.632835 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.633119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.633093 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.633579 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.633552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.634740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.634682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.635453 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.635433 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.733950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.733897 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f96c\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.736437 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.736364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token\") pod \"e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:15.782629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:15.782597 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:16.013542 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:16.013502 5091 generic.go:356] "Generic (PLEG): container finished" podID="04e51ec8-5426-4e79-a38f-c0ce22e3180b" containerID="8c6f726d6997757beaf77233e5c9ac7131e8daecb9083904dac90104c8ae4a3c" exitCode=0 Nov 02 22:01:16.014078 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:16.013662 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" event={"ID":"04e51ec8-5426-4e79-a38f-c0ce22e3180b","Type":"ContainerDied","Data":"8c6f726d6997757beaf77233e5c9ac7131e8daecb9083904dac90104c8ae4a3c"} Nov 02 22:01:16.294296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:16.294265 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj"] Nov 02 22:01:16.294882 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:01:16.294855 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11eed396_dc54_4649_8579_ae882d4bbac3.slice/crio-5b67d1d419da5ea74c136fcc2cf0f596710e51af162ced925cc572bf9353efe3 WatchSource:0}: Error finding container 5b67d1d419da5ea74c136fcc2cf0f596710e51af162ced925cc572bf9353efe3: Status 404 returned error can't find the container with id 5b67d1d419da5ea74c136fcc2cf0f596710e51af162ced925cc572bf9353efe3 Nov 02 22:01:16.298337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:16.298297 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:16.298438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:16.298382 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:16.298438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:16.298423 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:17.025039 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.024995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" event={"ID":"11eed396-dc54-4649-8579-ae882d4bbac3","Type":"ContainerStarted","Data":"24f3e322bcdde21764a23d660f25f7d2ae930dbd91a7c0d84b2b236e08739923"} Nov 02 22:01:17.025526 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.025065 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" event={"ID":"11eed396-dc54-4649-8579-ae882d4bbac3","Type":"ContainerStarted","Data":"5b67d1d419da5ea74c136fcc2cf0f596710e51af162ced925cc572bf9353efe3"} Nov 02 22:01:17.195147 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.195095 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podStartSLOduration=2.195078696 podStartE2EDuration="2.195078696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:17.188715568 +0000 UTC m=+1624.350810688" watchObservedRunningTime="2026-11-02 22:01:17.195078696 +0000 UTC m=+1624.357173816" Nov 02 22:01:17.226108 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.226084 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:17.277629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.277548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca\") pod \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " Nov 02 22:01:17.277831 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.277817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca\") pod \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " Nov 02 22:01:17.277908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.277892 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b\" (UniqueName: \"kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b\") pod \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\" (UID: \"04e51ec8-5426-4e79-a38f-c0ce22e3180b\") " Nov 02 22:01:17.278138 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.278112 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca" pod "04e51ec8-5426-4e79-a38f-c0ce22e3180b" (UID: "04e51ec8-5426-4e79-a38f-c0ce22e3180b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:17.278470 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.278447 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:17.279768 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.279742 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca" pod "04e51ec8-5426-4e79-a38f-c0ce22e3180b" (UID: "04e51ec8-5426-4e79-a38f-c0ce22e3180b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:17.279995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.279975 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b" pod "04e51ec8-5426-4e79-a38f-c0ce22e3180b" (UID: "04e51ec8-5426-4e79-a38f-c0ce22e3180b"). InnerVolumeSpecName "kube-api-access-49l2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:17.387752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.387705 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49l2b\" (UniqueName: \"kubernetes.io/projected/04e51ec8-5426-4e79-a38f-c0ce22e3180b-kube-api-access-49l2b\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:17.387752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.387746 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e51ec8-5426-4e79-a38f-c0ce22e3180b-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:17.783345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.783308 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:17.789263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:17.789238 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:18.035737 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.035641 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" Nov 02 22:01:18.035737 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.035687 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk" event={"ID":"04e51ec8-5426-4e79-a38f-c0ce22e3180b","Type":"ContainerDied","Data":"4598ec2c74f25ed632f9710bf7a23a60a89f9b7ceab464c4526f8cca9932c2e9"} Nov 02 22:01:18.035737 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.035737 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4598ec2c74f25ed632f9710bf7a23a60a89f9b7ceab464c4526f8cca9932c2e9" Nov 02 22:01:18.036205 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.035977 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:18.036925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.036907 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:01:18.049107 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.049086 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5"] Nov 02 22:01:18.053367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.053346 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008face6-sjws5"] Nov 02 22:01:18.302516 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.302421 5091 scope.go:122] "RemoveContainer" containerID="7c32d1027191e4653023d2e9f521d7cfb956c25dbc3212dbf055d93ce91b9a8c" Nov 02 22:01:18.313472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.313451 5091 scope.go:122] "RemoveContainer" containerID="1b5277c04c6b4146456058f3f7f03c5ada1e560d4667235060ef19c10b281aa0" Nov 02 22:01:18.323531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.323512 5091 scope.go:122] "RemoveContainer" containerID="6413083ff1cdd564d58decf10ec479133ee38901385d1872f0592376c6e44217" Nov 02 22:01:18.333169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.333149 5091 scope.go:122] "RemoveContainer" containerID="0f0e8266c96d717d41783b2235746c0d9707b5e8e92e0fc06064e9f225bc77bd" Nov 02 22:01:18.345812 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.345791 5091 scope.go:122] "RemoveContainer" containerID="13216b39182b17efd066f68663248d5033977993b42cd5ce0c1e923b6cb76c8e" Nov 02 22:01:18.355323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.355308 5091 scope.go:122] "RemoveContainer" containerID="34d08c8054adcb62f115e3427b18abbe1610d31083f179560e41b56d0d36d35e" Nov 02 22:01:18.364468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.364450 5091 scope.go:122] "RemoveContainer" containerID="d123db565d1c258aa29a19515692a2d6c2b1742a3487a909c875cb13655edfbb" Nov 02 22:01:18.373508 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.373494 5091 scope.go:122] "RemoveContainer" containerID="7cafa5af39a260d591ff47f4bad6f98414d7c90a50b68edd5cd0b03f34080394" Nov 02 22:01:18.382582 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.382558 5091 scope.go:122] "RemoveContainer" containerID="f966c76f6749a655bc13b1702b3c7daaa53df9bb9dc40fc7f37a149c55c2547b" Nov 02 22:01:18.978971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:18.978939 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a252fa78-d4b5-4852-bfff-41fa96f8699b" path="/var/lib/kubelet/pods/a252fa78-d4b5-4852-bfff-41fa96f8699b/volumes" Nov 02 22:01:19.053437 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.053391 5091 generic.go:356] "Generic (PLEG): container finished" podID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerID="797fd3bfcd7c837d01593b8aa1c3563305d41810f374eee10e667af36ffa42a3" exitCode=0 Nov 02 22:01:19.053934 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.053468 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" event={"ID":"5ec43bde-b150-4c1a-9b9c-19b241c69ede","Type":"ContainerDied","Data":"797fd3bfcd7c837d01593b8aa1c3563305d41810f374eee10e667af36ffa42a3"} Nov 02 22:01:19.055892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.055867 5091 generic.go:356] "Generic (PLEG): container finished" podID="84c11011-4005-4f80-94c3-454bc7f5de68" containerID="8bee2bed385041f76e11f9cec6bd703c4dafd6b1a22520f3e4403026c21b62f3" exitCode=0 Nov 02 22:01:19.056010 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.055934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" event={"ID":"84c11011-4005-4f80-94c3-454bc7f5de68","Type":"ContainerDied","Data":"8bee2bed385041f76e11f9cec6bd703c4dafd6b1a22520f3e4403026c21b62f3"} Nov 02 22:01:19.143123 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.143067 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:01:19.215444 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215415 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215487 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215541 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215566 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215617 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215650 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215733 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215769 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.215844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215812 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy\") pod \"84c11011-4005-4f80-94c3-454bc7f5de68\" (UID: \"84c11011-4005-4f80-94c3-454bc7f5de68\") " Nov 02 22:01:19.216102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215842 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.216102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.215856 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.216215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216106 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.216215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216126 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.216215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216173 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:19.216632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216609 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.216632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216630 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.216841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216640 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c11011-4005-4f80-94c3-454bc7f5de68-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.216841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216649 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.216841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.216657 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.217901 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.217873 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:19.218151 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.218130 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "kube-api-access-45p46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:19.218315 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.218297 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.218379 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.218305 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token" pod "84c11011-4005-4f80-94c3-454bc7f5de68" (UID: "84c11011-4005-4f80-94c3-454bc7f5de68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:19.318215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.318187 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45p46\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-kube-api-access-45p46\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.318215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.318215 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c11011-4005-4f80-94c3-454bc7f5de68-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.318433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.318224 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c11011-4005-4f80-94c3-454bc7f5de68-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.318433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.318233 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c11011-4005-4f80-94c3-454bc7f5de68-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.520735 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.520709 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:01:19.622876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.622844 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623029 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.622930 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623127 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623184 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623324 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623422 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.623493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623438 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623468 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623503 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:19.623637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623593 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623630 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket\") pod \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\" (UID: \"5ec43bde-b150-4c1a-9b9c-19b241c69ede\") " Nov 02 22:01:19.623822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623693 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.623964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.623940 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.624063 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.624020 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.624358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.624337 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.624429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.624363 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.624429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.624381 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.624429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.624395 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.624429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.624407 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.625245 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.625222 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "kube-api-access-jdbgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:19.625320 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.625270 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:19.625363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.625320 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:19.625593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.625580 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy" pod "5ec43bde-b150-4c1a-9b9c-19b241c69ede" (UID: "5ec43bde-b150-4c1a-9b9c-19b241c69ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.726143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.726064 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.726143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.726091 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.726143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.726102 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec43bde-b150-4c1a-9b9c-19b241c69ede-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.726143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:19.726111 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbgs\" (UniqueName: \"kubernetes.io/projected/5ec43bde-b150-4c1a-9b9c-19b241c69ede-kube-api-access-jdbgs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:20.067524 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.067435 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" Nov 02 22:01:20.067524 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.067464 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh" event={"ID":"84c11011-4005-4f80-94c3-454bc7f5de68","Type":"ContainerDied","Data":"cd235a6252c6d65623a5ab8909a37937514d2846900865378f13ba1fe3bfc05d"} Nov 02 22:01:20.068099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.067525 5091 scope.go:122] "RemoveContainer" containerID="8bee2bed385041f76e11f9cec6bd703c4dafd6b1a22520f3e4403026c21b62f3" Nov 02 22:01:20.069154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.069133 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" Nov 02 22:01:20.069301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.069161 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2" event={"ID":"5ec43bde-b150-4c1a-9b9c-19b241c69ede","Type":"ContainerDied","Data":"9a3d985294a6e94c650865c944fa6e8f431ec4bd9486102cec39d5b1b2b3f467"} Nov 02 22:01:20.080315 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.080299 5091 scope.go:122] "RemoveContainer" containerID="797fd3bfcd7c837d01593b8aa1c3563305d41810f374eee10e667af36ffa42a3" Nov 02 22:01:20.236279 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.236246 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh"] Nov 02 22:01:20.286015 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.285980 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-r57rh"] Nov 02 22:01:20.348367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.348301 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2"] Nov 02 22:01:20.358095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.358068 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-jnrq2"] Nov 02 22:01:20.978417 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.978388 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" path="/var/lib/kubelet/pods/5ec43bde-b150-4c1a-9b9c-19b241c69ede/volumes" Nov 02 22:01:20.978915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:20.978894 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" path="/var/lib/kubelet/pods/84c11011-4005-4f80-94c3-454bc7f5de68/volumes" Nov 02 22:01:25.214948 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:25.214908 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx"] Nov 02 22:01:25.264494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:25.264464 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4e90e53-fzcbx"] Nov 02 22:01:26.984335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:26.984291 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa03aac7-7678-490a-9d7b-dec6922e9125" path="/var/lib/kubelet/pods/aa03aac7-7678-490a-9d7b-dec6922e9125/volumes" Nov 02 22:01:30.045608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:30.045572 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t"] Nov 02 22:01:30.095033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:30.095002 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cff5fd-lbh2t"] Nov 02 22:01:30.981574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:01:30.981542 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ec18ee-a8f0-429b-b39c-1e4854607b5f" path="/var/lib/kubelet/pods/68ec18ee-a8f0-429b-b39c-1e4854607b5f/volumes" Nov 02 22:01:37.118380 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:01:37.118339 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:02:07.115261 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:02:07.115223 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:02:18.523653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:18.510810 5091 scope.go:122] "RemoveContainer" containerID="cb95ceb6c19d648cd9dff895dfc47c0b61a143e57c38ae1332b84ea9079a7ce4" Nov 02 22:02:18.574351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:18.573830 5091 scope.go:122] "RemoveContainer" containerID="b339749f1220fa6cc79d6a74b1ccb6bef338ad89a5ba745ef9807a4bc14f7504" Nov 02 22:02:18.621367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:18.621337 5091 scope.go:122] "RemoveContainer" containerID="c6544beb4cf6ab108a1721383137b4cbaf83dc5d83be5dfe70383a138d966bf1" Nov 02 22:02:18.723827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:18.723773 5091 scope.go:122] "RemoveContainer" containerID="f748bc15d79521557e46125d270f33fde352fb275beab591b9f29fa64f027e56" Nov 02 22:02:18.952771 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:18.950873 5091 scope.go:122] "RemoveContainer" containerID="6ce8452c010d4bc30cbb1ef83e0a3e8b819080d5b5b46b0f75f025981263ebec" Nov 02 22:02:19.050866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.050075 5091 scope.go:122] "RemoveContainer" containerID="61a3a3ba254b9c06230392a5099323d3793bf13e8641819100568188af34daf2" Nov 02 22:02:19.103932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.099724 5091 scope.go:122] "RemoveContainer" containerID="c1aaf0d1d3f1f9f682a0e29d6573b09f12e34ddbf7fb0cc0ae78318cd8df5efd" Nov 02 22:02:19.156619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.156469 5091 scope.go:122] "RemoveContainer" containerID="6e6f2622cefcbb04ad43b4679e17e944d053914a75985b1f159fdb5c12de3c17" Nov 02 22:02:19.215265 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.212664 5091 scope.go:122] "RemoveContainer" containerID="1edf107b46d5f8379f3b5d1449828622c12f5d11e0d2d647e0539d3ba1046f6b" Nov 02 22:02:19.271142 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.271012 5091 scope.go:122] "RemoveContainer" containerID="3c08e52c45f39398e4bdae59dd5f5114f36fdadffd63a0e58fb04c4e8d1f4f38" Nov 02 22:02:19.295637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.295299 5091 scope.go:122] "RemoveContainer" containerID="2d78c1d8785ed405bddf4fa09cdf2c03fdbb7d58028b60e0cbb959c041ff41fa" Nov 02 22:02:19.312402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.312297 5091 scope.go:122] "RemoveContainer" containerID="d005918f1de661ca37c70ddf0430fc4e717d631367eca9405a454af190219233" Nov 02 22:02:19.328755 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.328434 5091 scope.go:122] "RemoveContainer" containerID="a2f346eb3f3397195cefb0ff888959bd97bfc1017132f49e5c9a89ad8d717638" Nov 02 22:02:19.354832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.354804 5091 scope.go:122] "RemoveContainer" containerID="6226f6792bdd5c677fa0694b82941687cb0c9705b3f8240b9982fb286a836e9a" Nov 02 22:02:19.373288 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:19.373054 5091 scope.go:122] "RemoveContainer" containerID="e22abb510a975df72d739ba6736144464617b7f1597a499ace23532a9fa8811d" Nov 02 22:02:37.114971 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:02:37.114919 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:02:45.293126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:45.293088 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn"] Nov 02 22:02:45.293642 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:45.293355 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" containerID="cri-o://88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f" gracePeriod=30 Nov 02 22:02:49.078839 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:49.078803 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:02:49.079254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:49.078889 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:02:50.263619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.263543 5091 generic.go:356] "Generic (PLEG): container finished" podID="11eed396-dc54-4649-8579-ae882d4bbac3" containerID="24f3e322bcdde21764a23d660f25f7d2ae930dbd91a7c0d84b2b236e08739923" exitCode=0 Nov 02 22:02:50.264065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.263614 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" event={"ID":"11eed396-dc54-4649-8579-ae882d4bbac3","Type":"ContainerDied","Data":"24f3e322bcdde21764a23d660f25f7d2ae930dbd91a7c0d84b2b236e08739923"} Nov 02 22:02:50.264065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.264005 5091 scope.go:122] "RemoveContainer" containerID="24f3e322bcdde21764a23d660f25f7d2ae930dbd91a7c0d84b2b236e08739923" Nov 02 22:02:50.265538 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.265505 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:50.265658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.265588 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:50.265658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.265639 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:50.619513 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.619476 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 22:02:50.764690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.764464 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.764690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.764516 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.764966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.764699 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.764966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.764743 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.764966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.764828 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.764966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.764890 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.765193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.765019 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.765193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.765107 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.765193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.765165 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket\") pod \"435841da-c8b8-4060-85a5-346e9746144e\" (UID: \"435841da-c8b8-4060-85a5-346e9746144e\") " Nov 02 22:02:50.767087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.766659 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.767087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.766908 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.767901 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.767658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:50.768095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.768067 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.769228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.769203 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.770033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.770009 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.770173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.770157 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435841da-c8b8-4060-85a5-346e9746144e-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.770274 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.770261 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.770369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.770357 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.770455 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.770442 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.787097 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.787058 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "kube-api-access-7qp8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:50.787097 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.787071 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:50.787416 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.787394 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:50.788626 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.788572 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token" pod "435841da-c8b8-4060-85a5-346e9746144e" (UID: "435841da-c8b8-4060-85a5-346e9746144e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:50.872655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.872615 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435841da-c8b8-4060-85a5-346e9746144e-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.872655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.872652 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435841da-c8b8-4060-85a5-346e9746144e-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.872655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.872661 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.872655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:50.872688 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qp8m\" (UniqueName: \"kubernetes.io/projected/435841da-c8b8-4060-85a5-346e9746144e-kube-api-access-7qp8m\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.323315 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.323219 5091 generic.go:356] "Generic (PLEG): container finished" podID="435841da-c8b8-4060-85a5-346e9746144e" containerID="88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f" exitCode=0 Nov 02 22:02:51.324033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.324006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" event={"ID":"435841da-c8b8-4060-85a5-346e9746144e","Type":"ContainerDied","Data":"88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f"} Nov 02 22:02:51.324229 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.324203 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" event={"ID":"435841da-c8b8-4060-85a5-346e9746144e","Type":"ContainerDied","Data":"8acc69310d91b204aca89a5703b5f877c160a4b54f1699fbb08ce08d6caf8583"} Nov 02 22:02:51.324376 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.324361 5091 scope.go:122] "RemoveContainer" containerID="88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f" Nov 02 22:02:51.324644 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.324625 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn" Nov 02 22:02:51.345185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.344965 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" event={"ID":"11eed396-dc54-4649-8579-ae882d4bbac3","Type":"ContainerStarted","Data":"8d6e0837a8b31cb2a816655e3da584280a8b97141edc32c47770d165592c1b57"} Nov 02 22:02:51.356216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.354796 5091 scope.go:122] "RemoveContainer" containerID="88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f" Nov 02 22:02:51.357537 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:02:51.357505 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f\": container with ID starting with 88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f not found: ID does not exist" containerID="88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f" Nov 02 22:02:51.357634 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.357547 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f"} err="failed to get container status \"88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f\": rpc error: code = NotFound desc = could not find container \"88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f\": container with ID starting with 88d80fdf0aaab4b758a123f41f7e49ea11ced676cb6dd3b490a7c9afa74e904f not found: ID does not exist" Nov 02 22:02:51.630278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.630247 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn"] Nov 02 22:02:51.700693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.697824 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-jxhtn"] Nov 02 22:02:51.792697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.785046 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:02:51.793955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.793546 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:02:51.793955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:51.793612 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 02 22:02:52.787696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:52.785022 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:02:52.787696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:52.785114 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 02 22:02:53.011338 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:53.011300 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435841da-c8b8-4060-85a5-346e9746144e" path="/var/lib/kubelet/pods/435841da-c8b8-4060-85a5-346e9746144e/volumes" Nov 02 22:02:53.788217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:53.788177 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:02:53.788654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:53.788245 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 02 22:02:54.784696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.784634 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:02:54.787732 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.784965 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 02 22:02:54.996568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.996533 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt"] Nov 02 22:02:54.997420 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997396 5091 cpu_manager.go:414] "containerMap: removing container" podUID="04e51ec8-5426-4e79-a38f-c0ce22e3180b" containerName="revoke-keys" Nov 02 22:02:54.997539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997421 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e51ec8-5426-4e79-a38f-c0ce22e3180b" containerName="revoke-keys" Nov 02 22:02:54.997539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997508 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" Nov 02 22:02:54.997539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997517 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" Nov 02 22:02:54.997741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997544 5091 cpu_manager.go:414] "containerMap: removing container" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" Nov 02 22:02:54.997741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997551 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" Nov 02 22:02:54.997741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997571 5091 cpu_manager.go:414] "containerMap: removing container" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" Nov 02 22:02:54.997741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997579 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" Nov 02 22:02:54.997957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997763 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c11011-4005-4f80-94c3-454bc7f5de68" containerName="istio-proxy" Nov 02 22:02:54.997957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997788 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e51ec8-5426-4e79-a38f-c0ce22e3180b" containerName="revoke-keys" Nov 02 22:02:54.997957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997809 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec43bde-b150-4c1a-9b9c-19b241c69ede" containerName="istio-proxy" Nov 02 22:02:54.997957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:54.997821 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="435841da-c8b8-4060-85a5-346e9746144e" containerName="istio-proxy" Nov 02 22:02:55.001259 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.001237 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.046923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.046851 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt"] Nov 02 22:02:55.091241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.091163 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:02:55.091241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.091163 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:02:55.091241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.091177 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:02:55.102573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.102547 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:02:55.115738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.115708 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.115963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.115939 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.116078 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.116063 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vck\" (UniqueName: \"kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.217115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.217079 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.217324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.217193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.217324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.217291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vck\" (UniqueName: \"kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.217868 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.217840 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.219774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.219741 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.289749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.289663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vck\" (UniqueName: \"kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck\") pod \"maas-api-revoke-sub-4dd7c9c1-26jjt\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.315554 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.315488 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:55.730900 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:02:55.730864 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3605d602_28c2_438e_9fd7_3a570386dc96.slice/crio-0cf1ddea700c110ebc4159a6439b36c642d0bb5d9e0a60550e26d0cb0764b400 WatchSource:0}: Error finding container 0cf1ddea700c110ebc4159a6439b36c642d0bb5d9e0a60550e26d0cb0764b400: Status 404 returned error can't find the container with id 0cf1ddea700c110ebc4159a6439b36c642d0bb5d9e0a60550e26d0cb0764b400 Nov 02 22:02:55.731353 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.731328 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt"] Nov 02 22:02:55.789767 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.788567 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:02:55.789767 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:55.788638 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 02 22:02:56.447403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:56.447359 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" event={"ID":"3605d602-28c2-438e-9fd7-3a570386dc96","Type":"ContainerStarted","Data":"9090b008a43731ea17b29e9ff6afa67588455ff4b922815b810715c1f30c1149"} Nov 02 22:02:56.447863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:56.447427 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" event={"ID":"3605d602-28c2-438e-9fd7-3a570386dc96","Type":"ContainerStarted","Data":"0cf1ddea700c110ebc4159a6439b36c642d0bb5d9e0a60550e26d0cb0764b400"} Nov 02 22:02:56.566697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:56.562990 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" podStartSLOduration=2.562971637 podStartE2EDuration="2.562971637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:56.558758943 +0000 UTC m=+1723.720854064" watchObservedRunningTime="2026-11-02 22:02:56.562971637 +0000 UTC m=+1723.725066758" Nov 02 22:02:56.787944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:56.787917 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:02:56.788406 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:56.788393 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:02:56.789179 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:56.789158 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:02:57.456491 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:57.456457 5091 generic.go:356] "Generic (PLEG): container finished" podID="3605d602-28c2-438e-9fd7-3a570386dc96" containerID="9090b008a43731ea17b29e9ff6afa67588455ff4b922815b810715c1f30c1149" exitCode=0 Nov 02 22:02:57.457033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:57.456540 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" event={"ID":"3605d602-28c2-438e-9fd7-3a570386dc96","Type":"ContainerDied","Data":"9090b008a43731ea17b29e9ff6afa67588455ff4b922815b810715c1f30c1149"} Nov 02 22:02:58.621106 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.621081 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:58.675128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.675097 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca\") pod \"3605d602-28c2-438e-9fd7-3a570386dc96\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " Nov 02 22:02:58.675301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.675288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck\" (UniqueName: \"kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck\") pod \"3605d602-28c2-438e-9fd7-3a570386dc96\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " Nov 02 22:02:58.675355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.675339 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca\") pod \"3605d602-28c2-438e-9fd7-3a570386dc96\" (UID: \"3605d602-28c2-438e-9fd7-3a570386dc96\") " Nov 02 22:02:58.675773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.675748 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca" pod "3605d602-28c2-438e-9fd7-3a570386dc96" (UID: "3605d602-28c2-438e-9fd7-3a570386dc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:58.677119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.677096 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca" pod "3605d602-28c2-438e-9fd7-3a570386dc96" (UID: "3605d602-28c2-438e-9fd7-3a570386dc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:58.677436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.677411 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck" pod "3605d602-28c2-438e-9fd7-3a570386dc96" (UID: "3605d602-28c2-438e-9fd7-3a570386dc96"). InnerVolumeSpecName "kube-api-access-z7vck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:58.777310 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.777232 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7vck\" (UniqueName: \"kubernetes.io/projected/3605d602-28c2-438e-9fd7-3a570386dc96-kube-api-access-z7vck\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:58.777310 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.777263 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:58.777310 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:58.777277 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3605d602-28c2-438e-9fd7-3a570386dc96-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:59.471819 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:59.471781 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" event={"ID":"3605d602-28c2-438e-9fd7-3a570386dc96","Type":"ContainerDied","Data":"0cf1ddea700c110ebc4159a6439b36c642d0bb5d9e0a60550e26d0cb0764b400"} Nov 02 22:02:59.471819 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:59.471801 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt" Nov 02 22:02:59.472081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:02:59.471855 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf1ddea700c110ebc4159a6439b36c642d0bb5d9e0a60550e26d0cb0764b400" Nov 02 22:03:02.455492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:02.455453 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99"] Nov 02 22:03:02.456379 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:02.456305 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" containerID="cri-o://cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b" gracePeriod=30 Nov 02 22:03:07.113790 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:03:07.113751 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:03:07.435015 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.434981 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:03:07.435236 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.435055 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:03:07.750484 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.750462 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 22:03:07.906586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906550 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.906814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906656 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.906814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906730 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.906814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906804 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.907005 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906835 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.907005 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906902 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.907005 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.906934 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.907175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907077 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.907233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907180 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert\") pod \"05ec544d-1961-4f2a-a6e3-3d6122765837\" (UID: \"05ec544d-1961-4f2a-a6e3-3d6122765837\") " Nov 02 22:03:07.907295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907234 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907383 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907476 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907561 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907855 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907964 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.907989 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.908022 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:07.908061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.908037 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:07.909889 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.909864 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:07.910137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.910095 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:07.910260 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.910182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:03:07.910331 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:07.910313 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88" pod "05ec544d-1961-4f2a-a6e3-3d6122765837" (UID: "05ec544d-1961-4f2a-a6e3-3d6122765837"). InnerVolumeSpecName "kube-api-access-2ws88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:08.009957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.009884 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:08.009957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.009912 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ws88\" (UniqueName: \"kubernetes.io/projected/05ec544d-1961-4f2a-a6e3-3d6122765837-kube-api-access-2ws88\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:08.009957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.009923 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:08.009957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.009932 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec544d-1961-4f2a-a6e3-3d6122765837-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:08.009957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.009941 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec544d-1961-4f2a-a6e3-3d6122765837-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:03:08.549150 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.549115 5091 generic.go:356] "Generic (PLEG): container finished" podID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerID="cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b" exitCode=0 Nov 02 22:03:08.549565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.549212 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" Nov 02 22:03:08.549565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.549240 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" event={"ID":"05ec544d-1961-4f2a-a6e3-3d6122765837","Type":"ContainerDied","Data":"cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b"} Nov 02 22:03:08.549565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.549294 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99" event={"ID":"05ec544d-1961-4f2a-a6e3-3d6122765837","Type":"ContainerDied","Data":"8c6ee7d5862ef62cc4e176c5004ce5755318414b6f4917e6974e19badd06957f"} Nov 02 22:03:08.549565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.549316 5091 scope.go:122] "RemoveContainer" containerID="cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b" Nov 02 22:03:08.560966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.560950 5091 scope.go:122] "RemoveContainer" containerID="cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b" Nov 02 22:03:08.561244 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:03:08.561226 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b\": container with ID starting with cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b not found: ID does not exist" containerID="cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b" Nov 02 22:03:08.561294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.561251 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b"} err="failed to get container status \"cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b\": rpc error: code = NotFound desc = could not find container \"cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b\": container with ID starting with cc45081456044727e143801b56ea53a48681fb0c5c82c9925e9fb0e66bd9b68b not found: ID does not exist" Nov 02 22:03:08.764302 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.764274 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99"] Nov 02 22:03:08.814298 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.814225 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-kfx99"] Nov 02 22:03:08.978436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:08.978404 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" path="/var/lib/kubelet/pods/05ec544d-1961-4f2a-a6e3-3d6122765837/volumes" Nov 02 22:03:19.807528 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:03:19.807493 5091 scope.go:122] "RemoveContainer" containerID="111be525fe6fe2ea5d01fa0e2af89f58dd66eb6f85ef831ebfb845e7f11c378c" Nov 02 22:03:37.165065 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:03:37.164991 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:04:07.114603 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:04:07.114563 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:04:13.304364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:13.304333 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:04:13.316659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:13.316629 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:04:15.708770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:15.708729 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth"] Nov 02 22:04:15.709253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:15.708986 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" containerID="cri-o://ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba" gracePeriod=30 Nov 02 22:04:19.844195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:19.844158 5091 scope.go:122] "RemoveContainer" containerID="d60d217ef02fab451d4e4d95b09992799e74172bbc6f6c489438d5db279de94f" Nov 02 22:04:19.855082 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:19.855059 5091 scope.go:122] "RemoveContainer" containerID="284aaba3fb45843d2cf0ec0c278767217574a2d76c2d1ad585e56d1c42bc25c2" Nov 02 22:04:20.999794 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:20.999767 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 22:04:21.110135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110151 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110195 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110234 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110321 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110426 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110543 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110599 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy\") pod \"c3506f66-8b05-4c38-aff7-2451d20b7296\" (UID: \"c3506f66-8b05-4c38-aff7-2451d20b7296\") " Nov 02 22:04:21.110821 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110539 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:21.110821 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110618 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:21.110920 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.110843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:21.111168 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.111139 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:21.111587 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.111556 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:21.111729 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.111594 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.111729 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.111614 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.111729 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.111624 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3506f66-8b05-4c38-aff7-2451d20b7296-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.111729 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.111634 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.112636 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.112610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:21.112899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.112880 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:04:21.112972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.112885 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "kube-api-access-wb26t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:21.113204 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.113189 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy" pod "c3506f66-8b05-4c38-aff7-2451d20b7296" (UID: "c3506f66-8b05-4c38-aff7-2451d20b7296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:21.212714 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.212663 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb26t\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-kube-api-access-wb26t\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.212714 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.212711 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.212975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.212729 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.212975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.212740 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3506f66-8b05-4c38-aff7-2451d20b7296-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.212975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.212749 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3506f66-8b05-4c38-aff7-2451d20b7296-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:21.346518 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.346427 5091 generic.go:356] "Generic (PLEG): container finished" podID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerID="ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba" exitCode=0 Nov 02 22:04:21.346518 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.346492 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" Nov 02 22:04:21.346518 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.346499 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" event={"ID":"c3506f66-8b05-4c38-aff7-2451d20b7296","Type":"ContainerDied","Data":"ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba"} Nov 02 22:04:21.346841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.346600 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth" event={"ID":"c3506f66-8b05-4c38-aff7-2451d20b7296","Type":"ContainerDied","Data":"0b498a774e391892d4c2cf43d743a016795362fdd4932b1a25f273b7776303e6"} Nov 02 22:04:21.346841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.346633 5091 scope.go:122] "RemoveContainer" containerID="ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba" Nov 02 22:04:21.359145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.359127 5091 scope.go:122] "RemoveContainer" containerID="ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba" Nov 02 22:04:21.359402 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:04:21.359384 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba\": container with ID starting with ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba not found: ID does not exist" containerID="ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba" Nov 02 22:04:21.359456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.359408 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba"} err="failed to get container status \"ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba\": rpc error: code = NotFound desc = could not find container \"ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba\": container with ID starting with ec8239b4f93748c31e1a2d03ade407dc45ac0f4553ef9a84956da363a31f63ba not found: ID does not exist" Nov 02 22:04:21.510729 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.510696 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth"] Nov 02 22:04:21.515759 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:21.515732 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-5hzth"] Nov 02 22:04:22.980137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:22.980105 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" path="/var/lib/kubelet/pods/c3506f66-8b05-4c38-aff7-2451d20b7296/volumes" Nov 02 22:04:35.585959 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.585927 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7193-e5a62c"] Nov 02 22:04:35.586558 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586545 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" Nov 02 22:04:35.586602 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586559 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" Nov 02 22:04:35.586602 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586594 5091 cpu_manager.go:414] "containerMap: removing container" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" Nov 02 22:04:35.586602 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586599 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" Nov 02 22:04:35.586731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586606 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3605d602-28c2-438e-9fd7-3a570386dc96" containerName="revoke-keys" Nov 02 22:04:35.586731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586611 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3605d602-28c2-438e-9fd7-3a570386dc96" containerName="revoke-keys" Nov 02 22:04:35.586731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586720 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3506f66-8b05-4c38-aff7-2451d20b7296" containerName="istio-proxy" Nov 02 22:04:35.586825 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586733 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ec544d-1961-4f2a-a6e3-3d6122765837" containerName="istio-proxy" Nov 02 22:04:35.586825 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.586740 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3605d602-28c2-438e-9fd7-3a570386dc96" containerName="revoke-keys" Nov 02 22:04:35.590151 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.590133 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:35.644473 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.644440 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7193-e5a62c"] Nov 02 22:04:35.648052 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.648023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gh6\" (UniqueName: \"kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6\") pod \"test-curl-7193-e5a62c\" (UID: \"569b0d13-14f8-489d-b797-64b4ac0329b3\") " pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:35.689578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.689543 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m65jt\"" Nov 02 22:04:35.749826 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.749798 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gh6\" (UniqueName: \"kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6\") pod \"test-curl-7193-e5a62c\" (UID: \"569b0d13-14f8-489d-b797-64b4ac0329b3\") " pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:35.805897 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.805868 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gh6\" (UniqueName: \"kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6\") pod \"test-curl-7193-e5a62c\" (UID: \"569b0d13-14f8-489d-b797-64b4ac0329b3\") " pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:35.902984 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:35.902957 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:36.279610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:36.279577 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7193-e5a62c"] Nov 02 22:04:36.280518 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:04:36.280491 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod569b0d13_14f8_489d_b797_64b4ac0329b3.slice/crio-b79e519311c55fd2e55173f5c5f337c6665dfd17c5fb309493fc1e407aabc196 WatchSource:0}: Error finding container b79e519311c55fd2e55173f5c5f337c6665dfd17c5fb309493fc1e407aabc196: Status 404 returned error can't find the container with id b79e519311c55fd2e55173f5c5f337c6665dfd17c5fb309493fc1e407aabc196 Nov 02 22:04:36.493493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:36.493459 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7193-e5a62c" event={"ID":"569b0d13-14f8-489d-b797-64b4ac0329b3","Type":"ContainerStarted","Data":"2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4"} Nov 02 22:04:36.493726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:36.493508 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7193-e5a62c" event={"ID":"569b0d13-14f8-489d-b797-64b4ac0329b3","Type":"ContainerStarted","Data":"b79e519311c55fd2e55173f5c5f337c6665dfd17c5fb309493fc1e407aabc196"} Nov 02 22:04:36.609148 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:36.609034 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7193-e5a62c" podStartSLOduration=1.609015643 podStartE2EDuration="1.609015643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:36.605840196 +0000 UTC m=+1823.767935318" watchObservedRunningTime="2026-11-02 22:04:36.609015643 +0000 UTC m=+1823.771110768" Nov 02 22:04:37.135760 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:04:37.129283 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:04:38.153632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:38.153597 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7193-e5a62c"] Nov 02 22:04:38.159357 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:38.159321 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7193-e5a62c"] Nov 02 22:04:38.510879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:38.510758 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7193-e5a62c" podUID="569b0d13-14f8-489d-b797-64b4ac0329b3" containerName="test-curl-7193-e5a62c" containerID="cri-o://2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4" gracePeriod=2 Nov 02 22:04:38.561540 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:04:38.561500 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7193-e5a62c\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="569b0d13-14f8-489d-b797-64b4ac0329b3" pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:39.012290 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.012254 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974"] Nov 02 22:04:39.013330 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.013307 5091 cpu_manager.go:414] "containerMap: removing container" podUID="569b0d13-14f8-489d-b797-64b4ac0329b3" containerName="test-curl-7193-e5a62c" Nov 02 22:04:39.013409 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.013335 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b0d13-14f8-489d-b797-64b4ac0329b3" containerName="test-curl-7193-e5a62c" Nov 02 22:04:39.013502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.013489 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b0d13-14f8-489d-b797-64b4ac0329b3" containerName="test-curl-7193-e5a62c" Nov 02 22:04:39.017896 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.017871 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.071805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.071772 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974"] Nov 02 22:04:39.115924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.115895 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:39.116129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.116034 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:39.118979 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.118955 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:04:39.119115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.118975 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:39.168832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.168786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.169287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.168848 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.169287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.169050 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mg5\" (UniqueName: \"kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.270533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.270449 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.270533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.270488 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.270787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.270588 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mg5\" (UniqueName: \"kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.271253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.271232 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.272935 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.272910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.326930 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.326906 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mg5\" (UniqueName: \"kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.339787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.339761 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:39.714916 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:39.714879 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974"] Nov 02 22:04:39.715310 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:04:39.715283 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faf1b94_5bf3_46e6_b4ca_3d1c3ad19f88.slice/crio-3e6c12db99612b4abe26cf8997ff7ffef118e83bb203734e18d22b4b21c728dd WatchSource:0}: Error finding container 3e6c12db99612b4abe26cf8997ff7ffef118e83bb203734e18d22b4b21c728dd: Status 404 returned error can't find the container with id 3e6c12db99612b4abe26cf8997ff7ffef118e83bb203734e18d22b4b21c728dd Nov 02 22:04:40.527351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.527319 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" event={"ID":"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88","Type":"ContainerStarted","Data":"ed70ce54beb2dcbf2d184f17bcbb227afbe0e0552648ee2d39537dbd4bbdfc4d"} Nov 02 22:04:40.527692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.527400 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" event={"ID":"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88","Type":"ContainerStarted","Data":"3e6c12db99612b4abe26cf8997ff7ffef118e83bb203734e18d22b4b21c728dd"} Nov 02 22:04:40.642720 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.642652 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" podStartSLOduration=2.642638231 podStartE2EDuration="2.642638231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:40.640828849 +0000 UTC m=+1827.802923960" watchObservedRunningTime="2026-11-02 22:04:40.642638231 +0000 UTC m=+1827.804733351" Nov 02 22:04:40.731937 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:04:40.731889 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faf1b94_5bf3_46e6_b4ca_3d1c3ad19f88.slice/crio-ed70ce54beb2dcbf2d184f17bcbb227afbe0e0552648ee2d39537dbd4bbdfc4d.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:04:40.787295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.787267 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:40.892456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.892426 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6\" (UniqueName: \"kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6\") pod \"569b0d13-14f8-489d-b797-64b4ac0329b3\" (UID: \"569b0d13-14f8-489d-b797-64b4ac0329b3\") " Nov 02 22:04:40.894642 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.894611 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6" pod "569b0d13-14f8-489d-b797-64b4ac0329b3" (UID: "569b0d13-14f8-489d-b797-64b4ac0329b3"). InnerVolumeSpecName "kube-api-access-g2gh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:40.980986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.980958 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569b0d13-14f8-489d-b797-64b4ac0329b3" path="/var/lib/kubelet/pods/569b0d13-14f8-489d-b797-64b4ac0329b3/volumes" Nov 02 22:04:40.994666 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:40.994645 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2gh6\" (UniqueName: \"kubernetes.io/projected/569b0d13-14f8-489d-b797-64b4ac0329b3-kube-api-access-g2gh6\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.536461 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.536384 5091 generic.go:356] "Generic (PLEG): container finished" podID="6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" containerID="ed70ce54beb2dcbf2d184f17bcbb227afbe0e0552648ee2d39537dbd4bbdfc4d" exitCode=0 Nov 02 22:04:41.536950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.536460 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" event={"ID":"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88","Type":"ContainerDied","Data":"ed70ce54beb2dcbf2d184f17bcbb227afbe0e0552648ee2d39537dbd4bbdfc4d"} Nov 02 22:04:41.537745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.537724 5091 generic.go:356] "Generic (PLEG): container finished" podID="569b0d13-14f8-489d-b797-64b4ac0329b3" containerID="2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4" exitCode=137 Nov 02 22:04:41.537858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.537761 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-e5a62c" Nov 02 22:04:41.537858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.537815 5091 scope.go:122] "RemoveContainer" containerID="2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4" Nov 02 22:04:41.547545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.547529 5091 scope.go:122] "RemoveContainer" containerID="2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4" Nov 02 22:04:41.547857 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:04:41.547838 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4\": container with ID starting with 2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4 not found: ID does not exist" containerID="2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4" Nov 02 22:04:41.547919 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:41.547863 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4"} err="failed to get container status \"2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4\": rpc error: code = NotFound desc = could not find container \"2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4\": container with ID starting with 2a6ad5f4f18dd0b55679aa16048831c469f3f2f0515e465e1d2a7874734e34d4 not found: ID does not exist" Nov 02 22:04:42.712115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.712088 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:42.819452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.819375 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca\") pod \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " Nov 02 22:04:42.819610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.819534 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca\") pod \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " Nov 02 22:04:42.819610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.819570 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5\" (UniqueName: \"kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5\") pod \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\" (UID: \"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88\") " Nov 02 22:04:42.819805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.819779 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca" pod "6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" (UID: "6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:42.820203 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.820180 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:42.821727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.821701 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca" pod "6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" (UID: "6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:42.821809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.821741 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5" pod "6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" (UID: "6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88"). InnerVolumeSpecName "kube-api-access-x2mg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:42.933219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.933177 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:42.933219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:42.933219 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2mg5\" (UniqueName: \"kubernetes.io/projected/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88-kube-api-access-x2mg5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:04:43.554276 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:43.554248 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" Nov 02 22:04:43.554452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:43.554284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974" event={"ID":"6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88","Type":"ContainerDied","Data":"3e6c12db99612b4abe26cf8997ff7ffef118e83bb203734e18d22b4b21c728dd"} Nov 02 22:04:43.554452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:43.554333 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6c12db99612b4abe26cf8997ff7ffef118e83bb203734e18d22b4b21c728dd" Nov 02 22:04:56.439083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.439006 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f"] Nov 02 22:04:56.439700 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.439659 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" containerName="revoke-keys" Nov 02 22:04:56.439700 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.439690 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" containerName="revoke-keys" Nov 02 22:04:56.439806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.439799 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" containerName="revoke-keys" Nov 02 22:04:56.443735 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.443718 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.500366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.500340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f"] Nov 02 22:04:56.544464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.544436 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:04:56.544781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.544765 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:04:56.577617 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.577587 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhf5r\" (UniqueName: \"kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.577813 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.577793 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.577863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.577823 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.582698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.582647 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:04:56.582698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.582662 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:04:56.679398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.679367 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.679398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.679396 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.679596 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.679520 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhf5r\" (UniqueName: \"kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.680158 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.680135 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.681858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.681840 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.735946 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.735887 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhf5r\" (UniqueName: \"kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r\") pod \"maas-api-revoke-sub-484614ae-r6f8f\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:56.757249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:56.757227 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:04:57.134527 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:04:57.134498 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1357ea_b0e5_407a_8d86_3082636a24f0.slice/crio-6711efb26f4a71ff2153b91cd234cf2a13f857354ee657aa549b4c4306cebb50 WatchSource:0}: Error finding container 6711efb26f4a71ff2153b91cd234cf2a13f857354ee657aa549b4c4306cebb50: Status 404 returned error can't find the container with id 6711efb26f4a71ff2153b91cd234cf2a13f857354ee657aa549b4c4306cebb50 Nov 02 22:04:57.134657 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:57.134531 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f"] Nov 02 22:04:57.799201 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:57.799114 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" event={"ID":"1f1357ea-b0e5-407a-8d86-3082636a24f0","Type":"ContainerStarted","Data":"c26e3a4c4b0d80330f19ecf258b33140c2f6015aec549e72139a79644ea80007"} Nov 02 22:04:57.799201 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:57.799179 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" event={"ID":"1f1357ea-b0e5-407a-8d86-3082636a24f0","Type":"ContainerStarted","Data":"6711efb26f4a71ff2153b91cd234cf2a13f857354ee657aa549b4c4306cebb50"} Nov 02 22:04:57.919887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:57.919838 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" podStartSLOduration=1.919824384 podStartE2EDuration="1.919824384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:57.915136324 +0000 UTC m=+1845.077231444" watchObservedRunningTime="2026-11-02 22:04:57.919824384 +0000 UTC m=+1845.081919503" Nov 02 22:04:58.808636 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:58.808600 5091 generic.go:356] "Generic (PLEG): container finished" podID="1f1357ea-b0e5-407a-8d86-3082636a24f0" containerID="c26e3a4c4b0d80330f19ecf258b33140c2f6015aec549e72139a79644ea80007" exitCode=0 Nov 02 22:04:58.809092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:58.808748 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" event={"ID":"1f1357ea-b0e5-407a-8d86-3082636a24f0","Type":"ContainerDied","Data":"c26e3a4c4b0d80330f19ecf258b33140c2f6015aec549e72139a79644ea80007"} Nov 02 22:04:59.977173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:04:59.977146 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:05:00.108861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.108783 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca\") pod \"1f1357ea-b0e5-407a-8d86-3082636a24f0\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " Nov 02 22:05:00.109017 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.108966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r\" (UniqueName: \"kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r\") pod \"1f1357ea-b0e5-407a-8d86-3082636a24f0\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " Nov 02 22:05:00.109159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.109137 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca\") pod \"1f1357ea-b0e5-407a-8d86-3082636a24f0\" (UID: \"1f1357ea-b0e5-407a-8d86-3082636a24f0\") " Nov 02 22:05:00.109236 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.109160 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca" pod "1f1357ea-b0e5-407a-8d86-3082636a24f0" (UID: "1f1357ea-b0e5-407a-8d86-3082636a24f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:05:00.109865 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.109842 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:00.170357 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.169399 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r" pod "1f1357ea-b0e5-407a-8d86-3082636a24f0" (UID: "1f1357ea-b0e5-407a-8d86-3082636a24f0"). InnerVolumeSpecName "kube-api-access-jhf5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:00.170357 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.170310 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca" pod "1f1357ea-b0e5-407a-8d86-3082636a24f0" (UID: "1f1357ea-b0e5-407a-8d86-3082636a24f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:00.211515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.211484 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhf5r\" (UniqueName: \"kubernetes.io/projected/1f1357ea-b0e5-407a-8d86-3082636a24f0-kube-api-access-jhf5r\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:00.211515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.211512 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1357ea-b0e5-407a-8d86-3082636a24f0-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:00.824931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.824901 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" Nov 02 22:05:00.825124 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.824929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f" event={"ID":"1f1357ea-b0e5-407a-8d86-3082636a24f0","Type":"ContainerDied","Data":"6711efb26f4a71ff2153b91cd234cf2a13f857354ee657aa549b4c4306cebb50"} Nov 02 22:05:00.825124 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:00.824979 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6711efb26f4a71ff2153b91cd234cf2a13f857354ee657aa549b4c4306cebb50" Nov 02 22:05:07.115330 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:05:07.115293 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:05:13.888643 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.888604 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk"] Nov 02 22:05:13.889754 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.889480 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1f1357ea-b0e5-407a-8d86-3082636a24f0" containerName="revoke-keys" Nov 02 22:05:13.889754 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.889498 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1357ea-b0e5-407a-8d86-3082636a24f0" containerName="revoke-keys" Nov 02 22:05:13.889754 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.889624 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1357ea-b0e5-407a-8d86-3082636a24f0" containerName="revoke-keys" Nov 02 22:05:13.893315 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.893297 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:13.949087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.949021 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk"] Nov 02 22:05:13.972512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.972477 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2ks\" (UniqueName: \"kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:13.972660 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.972604 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:13.972660 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.972623 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:13.992496 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.992471 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:05:13.992639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.992494 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:05:13.994414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.994395 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:05:13.994531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:13.994414 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:05:14.074160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.074121 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.074160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.074160 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.074629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.074598 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2ks\" (UniqueName: \"kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.074873 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.074848 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.076940 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.076918 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.130922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.130894 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2ks\" (UniqueName: \"kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks\") pod \"maas-api-revoke-sub-ee35b83a-ttmnk\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.208974 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.208913 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:14.588358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.588329 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk"] Nov 02 22:05:14.589123 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:05:14.589081 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe620b5_b6cf_444b_8fd7_f3e5d021e317.slice/crio-42de3fcf6bfc6342d92f15e6457dd8d5da4a15f83906c2fd5af6f63c7fa7079c WatchSource:0}: Error finding container 42de3fcf6bfc6342d92f15e6457dd8d5da4a15f83906c2fd5af6f63c7fa7079c: Status 404 returned error can't find the container with id 42de3fcf6bfc6342d92f15e6457dd8d5da4a15f83906c2fd5af6f63c7fa7079c Nov 02 22:05:14.977492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.977460 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" event={"ID":"9fe620b5-b6cf-444b-8fd7-f3e5d021e317","Type":"ContainerStarted","Data":"a7c9a7b798db2c58817d7fad4317e47691a7b4c29d2d29e7bd806f0c3bdd3621"} Nov 02 22:05:14.977877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:14.977509 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" event={"ID":"9fe620b5-b6cf-444b-8fd7-f3e5d021e317","Type":"ContainerStarted","Data":"42de3fcf6bfc6342d92f15e6457dd8d5da4a15f83906c2fd5af6f63c7fa7079c"} Nov 02 22:05:15.089991 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:15.089942 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" podStartSLOduration=2.0899264730000002 podStartE2EDuration="2.089926473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:15.086982729 +0000 UTC m=+1862.249077849" watchObservedRunningTime="2026-11-02 22:05:15.089926473 +0000 UTC m=+1862.252021592" Nov 02 22:05:15.982751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:15.982712 5091 generic.go:356] "Generic (PLEG): container finished" podID="9fe620b5-b6cf-444b-8fd7-f3e5d021e317" containerID="a7c9a7b798db2c58817d7fad4317e47691a7b4c29d2d29e7bd806f0c3bdd3621" exitCode=0 Nov 02 22:05:15.983161 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:15.982844 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" event={"ID":"9fe620b5-b6cf-444b-8fd7-f3e5d021e317","Type":"ContainerDied","Data":"a7c9a7b798db2c58817d7fad4317e47691a7b4c29d2d29e7bd806f0c3bdd3621"} Nov 02 22:05:17.166638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.166610 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:17.217963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.217928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks\" (UniqueName: \"kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks\") pod \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " Nov 02 22:05:17.218156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.218007 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca\") pod \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " Nov 02 22:05:17.218223 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.218187 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca\") pod \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\" (UID: \"9fe620b5-b6cf-444b-8fd7-f3e5d021e317\") " Nov 02 22:05:17.218816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.218785 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca" pod "9fe620b5-b6cf-444b-8fd7-f3e5d021e317" (UID: "9fe620b5-b6cf-444b-8fd7-f3e5d021e317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:05:17.220963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.220931 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks" pod "9fe620b5-b6cf-444b-8fd7-f3e5d021e317" (UID: "9fe620b5-b6cf-444b-8fd7-f3e5d021e317"). InnerVolumeSpecName "kube-api-access-tc2ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:17.221266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.221248 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca" pod "9fe620b5-b6cf-444b-8fd7-f3e5d021e317" (UID: "9fe620b5-b6cf-444b-8fd7-f3e5d021e317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:17.319915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.319847 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2ks\" (UniqueName: \"kubernetes.io/projected/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-kube-api-access-tc2ks\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:17.319915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.319874 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:17.319915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.319885 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe620b5-b6cf-444b-8fd7-f3e5d021e317-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:17.999043 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.999013 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" Nov 02 22:05:17.999043 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.999035 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk" event={"ID":"9fe620b5-b6cf-444b-8fd7-f3e5d021e317","Type":"ContainerDied","Data":"42de3fcf6bfc6342d92f15e6457dd8d5da4a15f83906c2fd5af6f63c7fa7079c"} Nov 02 22:05:17.999312 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:17.999072 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42de3fcf6bfc6342d92f15e6457dd8d5da4a15f83906c2fd5af6f63c7fa7079c" Nov 02 22:05:19.886773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:19.886743 5091 scope.go:122] "RemoveContainer" containerID="cc659fb48632ec6a2d61aa26c0e2b2d0cd32c34c1df2353b1124f348fb63ff2b" Nov 02 22:05:19.900364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:19.900339 5091 scope.go:122] "RemoveContainer" containerID="e2160971d6520ab7c6ebf1611e1b35cacf9fe40726f587a066615940618acda7" Nov 02 22:05:26.046468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:26.046427 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg"] Nov 02 22:05:26.096337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:26.096306 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b1c8da-gtpgg"] Nov 02 22:05:26.978044 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:05:26.978002 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f" path="/var/lib/kubelet/pods/ad9bfd17-0af7-46d7-8aa5-8ee9c897d02f/volumes" Nov 02 22:05:37.115571 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:05:37.115534 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:06:07.113537 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:07.113499 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:06:18.047031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:18.046999 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk"] Nov 02 22:06:18.097213 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:18.097184 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb83ef0-ql2wk"] Nov 02 22:06:18.978159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:18.978125 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e51ec8-5426-4e79-a38f-c0ce22e3180b" path="/var/lib/kubelet/pods/04e51ec8-5426-4e79-a38f-c0ce22e3180b/volumes" Nov 02 22:06:37.117285 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:37.117200 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:06:37.892110 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.892074 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8"] Nov 02 22:06:37.892792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.892777 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9fe620b5-b6cf-444b-8fd7-f3e5d021e317" containerName="revoke-keys" Nov 02 22:06:37.892857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.892794 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe620b5-b6cf-444b-8fd7-f3e5d021e317" containerName="revoke-keys" Nov 02 22:06:37.892909 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.892899 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe620b5-b6cf-444b-8fd7-f3e5d021e317" containerName="revoke-keys" Nov 02 22:06:37.896513 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.896489 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:37.952966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.952940 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8"] Nov 02 22:06:37.977953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.977914 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:37.978112 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.978089 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:37.978240 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.978217 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrzjv\" (UniqueName: \"kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:37.997616 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.997592 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:06:37.997616 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:37.997605 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:06:38.023726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.023687 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:06:38.047316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.047296 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:06:38.080080 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.080056 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.080178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.080115 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.080178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.080162 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrzjv\" (UniqueName: \"kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.080777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.080756 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.082525 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.082504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.136506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.136477 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrzjv\" (UniqueName: \"kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv\") pod \"maas-api-revoke-sub-9122c16b-vfhl8\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.210696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.210595 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:38.594466 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:06:38.594430 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34125d68_c999_4549_90b0_80630fcf8407.slice/crio-7a7e4f6d2bcd02845dbde0c9c3f90df41edba985b5e4478b6b1cfbbb98343fa0 WatchSource:0}: Error finding container 7a7e4f6d2bcd02845dbde0c9c3f90df41edba985b5e4478b6b1cfbbb98343fa0: Status 404 returned error can't find the container with id 7a7e4f6d2bcd02845dbde0c9c3f90df41edba985b5e4478b6b1cfbbb98343fa0 Nov 02 22:06:38.600352 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.600326 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8"] Nov 02 22:06:38.854286 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:38.854200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" event={"ID":"34125d68-c999-4549-90b0-80630fcf8407","Type":"ContainerStarted","Data":"7a7e4f6d2bcd02845dbde0c9c3f90df41edba985b5e4478b6b1cfbbb98343fa0"} Nov 02 22:06:39.864129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:39.864031 5091 generic.go:356] "Generic (PLEG): container finished" podID="34125d68-c999-4549-90b0-80630fcf8407" containerID="b3540120a820d5e5cbd247e9058e2f5a53a33abdcbbd4de1a103639ee3da3d4d" exitCode=0 Nov 02 22:06:39.864129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:39.864082 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" event={"ID":"34125d68-c999-4549-90b0-80630fcf8407","Type":"ContainerDied","Data":"b3540120a820d5e5cbd247e9058e2f5a53a33abdcbbd4de1a103639ee3da3d4d"} Nov 02 22:06:41.036881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.036853 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:41.230301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.230210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv\" (UniqueName: \"kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv\") pod \"34125d68-c999-4549-90b0-80630fcf8407\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " Nov 02 22:06:41.230301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.230278 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca\") pod \"34125d68-c999-4549-90b0-80630fcf8407\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " Nov 02 22:06:41.230556 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.230492 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca\") pod \"34125d68-c999-4549-90b0-80630fcf8407\" (UID: \"34125d68-c999-4549-90b0-80630fcf8407\") " Nov 02 22:06:41.230951 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.230920 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca" pod "34125d68-c999-4549-90b0-80630fcf8407" (UID: "34125d68-c999-4549-90b0-80630fcf8407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:41.231217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.231200 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34125d68-c999-4549-90b0-80630fcf8407-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:41.232639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.232616 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca" pod "34125d68-c999-4549-90b0-80630fcf8407" (UID: "34125d68-c999-4549-90b0-80630fcf8407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:41.233000 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.232973 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv" pod "34125d68-c999-4549-90b0-80630fcf8407" (UID: "34125d68-c999-4549-90b0-80630fcf8407"). InnerVolumeSpecName "kube-api-access-zrzjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:41.333112 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.333082 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrzjv\" (UniqueName: \"kubernetes.io/projected/34125d68-c999-4549-90b0-80630fcf8407-kube-api-access-zrzjv\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:41.333112 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.333111 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34125d68-c999-4549-90b0-80630fcf8407-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:41.880360 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.880330 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" Nov 02 22:06:41.880561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.880365 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8" event={"ID":"34125d68-c999-4549-90b0-80630fcf8407","Type":"ContainerDied","Data":"7a7e4f6d2bcd02845dbde0c9c3f90df41edba985b5e4478b6b1cfbbb98343fa0"} Nov 02 22:06:41.880561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:41.880448 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a7e4f6d2bcd02845dbde0c9c3f90df41edba985b5e4478b6b1cfbbb98343fa0" Nov 02 22:06:42.940976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.940944 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7202-70e147"] Nov 02 22:06:42.941601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.941586 5091 cpu_manager.go:414] "containerMap: removing container" podUID="34125d68-c999-4549-90b0-80630fcf8407" containerName="revoke-keys" Nov 02 22:06:42.941647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.941602 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34125d68-c999-4549-90b0-80630fcf8407" containerName="revoke-keys" Nov 02 22:06:42.941748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.941738 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="34125d68-c999-4549-90b0-80630fcf8407" containerName="revoke-keys" Nov 02 22:06:42.946583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.946566 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:42.952922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.952890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hvg\" (UniqueName: \"kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg\") pod \"test-curl-7202-70e147\" (UID: \"992b83ff-1744-468e-b4ab-c87800c42f7e\") " pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:42.998656 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:42.998625 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-70e147"] Nov 02 22:06:43.042808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.042778 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m65jt\"" Nov 02 22:06:43.054932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.054899 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hvg\" (UniqueName: \"kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg\") pod \"test-curl-7202-70e147\" (UID: \"992b83ff-1744-468e-b4ab-c87800c42f7e\") " pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:43.110869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.110840 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hvg\" (UniqueName: \"kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg\") pod \"test-curl-7202-70e147\" (UID: \"992b83ff-1744-468e-b4ab-c87800c42f7e\") " pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:43.260315 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.260240 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:43.635190 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.635091 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-70e147"] Nov 02 22:06:43.896583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.896485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-70e147" event={"ID":"992b83ff-1744-468e-b4ab-c87800c42f7e","Type":"ContainerStarted","Data":"b9bb850c0d218521d7b62079132997a099565f19179b4d36e038f1c3026a78dd"} Nov 02 22:06:43.896583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:43.896533 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-70e147" event={"ID":"992b83ff-1744-468e-b4ab-c87800c42f7e","Type":"ContainerStarted","Data":"eb9a1cf2572d77a028870c6c2e3689440bc44a5d09a2a2dfff80caec592fa139"} Nov 02 22:06:44.007788 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:44.007256 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7202-70e147" podStartSLOduration=2.007239623 podStartE2EDuration="2.007239623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:44.006240798 +0000 UTC m=+1951.168335920" watchObservedRunningTime="2026-11-02 22:06:44.007239623 +0000 UTC m=+1951.169334748" Nov 02 22:06:45.364320 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.364284 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7202-70e147"] Nov 02 22:06:45.369835 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.369806 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7202-70e147"] Nov 02 22:06:45.516529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.516497 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7202-2edeb5"] Nov 02 22:06:45.517311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.517291 5091 cpu_manager.go:414] "containerMap: removing container" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" containerName="test-curl-7202-70e147" Nov 02 22:06:45.517406 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.517312 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" containerName="test-curl-7202-70e147" Nov 02 22:06:45.517475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.517442 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" containerName="test-curl-7202-70e147" Nov 02 22:06:45.521031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.521015 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:45.575889 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.575865 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-2edeb5"] Nov 02 22:06:45.589228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.589201 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25ttn\" (UniqueName: \"kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn\") pod \"test-curl-7202-2edeb5\" (UID: \"3ff91206-fa5c-42b5-bcc2-1ccf6c754530\") " pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:45.690986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.690953 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25ttn\" (UniqueName: \"kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn\") pod \"test-curl-7202-2edeb5\" (UID: \"3ff91206-fa5c-42b5-bcc2-1ccf6c754530\") " pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:45.753630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.753593 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25ttn\" (UniqueName: \"kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn\") pod \"test-curl-7202-2edeb5\" (UID: \"3ff91206-fa5c-42b5-bcc2-1ccf6c754530\") " pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:45.837195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.836316 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:45.918719 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:45.917558 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7202-70e147" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" containerName="test-curl-7202-70e147" containerID="cri-o://b9bb850c0d218521d7b62079132997a099565f19179b4d36e038f1c3026a78dd" gracePeriod=2 Nov 02 22:06:45.968625 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:45.968540 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-70e147\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:46.215259 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:46.215228 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-2edeb5"] Nov 02 22:06:46.216739 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:06:46.216707 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ff91206_fa5c_42b5_bcc2_1ccf6c754530.slice/crio-a54933dece0bcd07f3f670aef051fabc825aec82c279433f075013129b69aec4 WatchSource:0}: Error finding container a54933dece0bcd07f3f670aef051fabc825aec82c279433f075013129b69aec4: Status 404 returned error can't find the container with id a54933dece0bcd07f3f670aef051fabc825aec82c279433f075013129b69aec4 Nov 02 22:06:46.927344 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:46.927309 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-2edeb5" event={"ID":"3ff91206-fa5c-42b5-bcc2-1ccf6c754530","Type":"ContainerStarted","Data":"4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c"} Nov 02 22:06:46.927763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:46.927354 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-2edeb5" event={"ID":"3ff91206-fa5c-42b5-bcc2-1ccf6c754530","Type":"ContainerStarted","Data":"a54933dece0bcd07f3f670aef051fabc825aec82c279433f075013129b69aec4"} Nov 02 22:06:46.977907 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:46.977868 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-70e147\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:47.092348 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:47.092294 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7202-2edeb5" podStartSLOduration=2.092279729 podStartE2EDuration="2.092279729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:47.089537281 +0000 UTC m=+1954.251632416" watchObservedRunningTime="2026-11-02 22:06:47.092279729 +0000 UTC m=+1954.254374849" Nov 02 22:06:47.936984 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:47.936951 5091 generic.go:356] "Generic (PLEG): container finished" podID="992b83ff-1744-468e-b4ab-c87800c42f7e" containerID="b9bb850c0d218521d7b62079132997a099565f19179b4d36e038f1c3026a78dd" exitCode=137 Nov 02 22:06:48.142365 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.142340 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:48.226974 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.226898 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg\" (UniqueName: \"kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg\") pod \"992b83ff-1744-468e-b4ab-c87800c42f7e\" (UID: \"992b83ff-1744-468e-b4ab-c87800c42f7e\") " Nov 02 22:06:48.229171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.229147 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg" pod "992b83ff-1744-468e-b4ab-c87800c42f7e" (UID: "992b83ff-1744-468e-b4ab-c87800c42f7e"). InnerVolumeSpecName "kube-api-access-x2hvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:48.329053 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.329029 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hvg\" (UniqueName: \"kubernetes.io/projected/992b83ff-1744-468e-b4ab-c87800c42f7e-kube-api-access-x2hvg\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:48.553884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.553798 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7202-2edeb5"] Nov 02 22:06:48.558456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.558426 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7202-2edeb5"] Nov 02 22:06:48.702450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.702418 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7202-a85cb4"] Nov 02 22:06:48.703093 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.703078 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" containerName="test-curl-7202-2edeb5" Nov 02 22:06:48.703146 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.703095 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" containerName="test-curl-7202-2edeb5" Nov 02 22:06:48.703217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.703208 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" containerName="test-curl-7202-2edeb5" Nov 02 22:06:48.706777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.706755 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:48.760785 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.760758 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-a85cb4"] Nov 02 22:06:48.835855 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.835780 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfb8\" (UniqueName: \"kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8\") pod \"test-curl-7202-a85cb4\" (UID: \"9fd6f5aa-4de0-4a34-af9f-0061392f0229\") " pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:48.937093 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.937057 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfb8\" (UniqueName: \"kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8\") pod \"test-curl-7202-a85cb4\" (UID: \"9fd6f5aa-4de0-4a34-af9f-0061392f0229\") " pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:48.944928 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.944904 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-70e147" Nov 02 22:06:48.945068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.944957 5091 scope.go:122] "RemoveContainer" containerID="b9bb850c0d218521d7b62079132997a099565f19179b4d36e038f1c3026a78dd" Nov 02 22:06:48.945126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.945104 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7202-2edeb5" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" containerName="test-curl-7202-2edeb5" containerID="cri-o://4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c" gracePeriod=2 Nov 02 22:06:48.978483 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.978454 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992b83ff-1744-468e-b4ab-c87800c42f7e" path="/var/lib/kubelet/pods/992b83ff-1744-468e-b4ab-c87800c42f7e/volumes" Nov 02 22:06:48.993560 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:48.993527 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfb8\" (UniqueName: \"kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8\") pod \"test-curl-7202-a85cb4\" (UID: \"9fd6f5aa-4de0-4a34-af9f-0061392f0229\") " pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:49.007979 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:49.007932 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-2edeb5\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:49.019156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:49.019133 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:49.023049 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:49.023023 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod992b83ff_1744_468e_b4ab_c87800c42f7e.slice/crio-eb9a1cf2572d77a028870c6c2e3689440bc44a5d09a2a2dfff80caec592fa139\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod992b83ff_1744_468e_b4ab_c87800c42f7e.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:06:49.406037 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:49.406008 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-a85cb4"] Nov 02 22:06:49.407054 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:06:49.407022 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fd6f5aa_4de0_4a34_af9f_0061392f0229.slice/crio-6e5a3bb431a26ae37a0f1dcc564d32077a91dbfa9b5ba5f3eac2f043f4f208f5 WatchSource:0}: Error finding container 6e5a3bb431a26ae37a0f1dcc564d32077a91dbfa9b5ba5f3eac2f043f4f208f5: Status 404 returned error can't find the container with id 6e5a3bb431a26ae37a0f1dcc564d32077a91dbfa9b5ba5f3eac2f043f4f208f5 Nov 02 22:06:49.953718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:49.953684 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-a85cb4" event={"ID":"9fd6f5aa-4de0-4a34-af9f-0061392f0229","Type":"ContainerStarted","Data":"6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba"} Nov 02 22:06:49.954128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:49.953731 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-a85cb4" event={"ID":"9fd6f5aa-4de0-4a34-af9f-0061392f0229","Type":"ContainerStarted","Data":"6e5a3bb431a26ae37a0f1dcc564d32077a91dbfa9b5ba5f3eac2f043f4f208f5"} Nov 02 22:06:50.066936 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:50.066883 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7202-a85cb4" podStartSLOduration=2.06686891 podStartE2EDuration="2.06686891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:50.063964404 +0000 UTC m=+1957.226059523" watchObservedRunningTime="2026-11-02 22:06:50.06686891 +0000 UTC m=+1957.228964029" Nov 02 22:06:51.423431 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.423406 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:51.517314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.517287 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn\" (UniqueName: \"kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn\") pod \"3ff91206-fa5c-42b5-bcc2-1ccf6c754530\" (UID: \"3ff91206-fa5c-42b5-bcc2-1ccf6c754530\") " Nov 02 22:06:51.519368 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.519343 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn" pod "3ff91206-fa5c-42b5-bcc2-1ccf6c754530" (UID: "3ff91206-fa5c-42b5-bcc2-1ccf6c754530"). InnerVolumeSpecName "kube-api-access-25ttn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:51.619354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.619329 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25ttn\" (UniqueName: \"kubernetes.io/projected/3ff91206-fa5c-42b5-bcc2-1ccf6c754530-kube-api-access-25ttn\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:51.630074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.630040 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7202-a85cb4"] Nov 02 22:06:51.634198 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.634168 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7202-a85cb4"] Nov 02 22:06:51.779904 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.779865 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7202-401fe2"] Nov 02 22:06:51.780743 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.780724 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" containerName="test-curl-7202-a85cb4" Nov 02 22:06:51.780806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.780745 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" containerName="test-curl-7202-a85cb4" Nov 02 22:06:51.780891 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.780882 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" containerName="test-curl-7202-a85cb4" Nov 02 22:06:51.784381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.784362 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:51.839890 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.839861 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-401fe2"] Nov 02 22:06:51.923414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.923335 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dw7\" (UniqueName: \"kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7\") pod \"test-curl-7202-401fe2\" (UID: \"c28355f0-4117-4777-ae12-8c392dcd75e2\") " pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:51.971183 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.971147 5091 generic.go:356] "Generic (PLEG): container finished" podID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" containerID="4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c" exitCode=137 Nov 02 22:06:51.971183 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.971177 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:51.971408 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.971238 5091 scope.go:122] "RemoveContainer" containerID="4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c" Nov 02 22:06:51.971567 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.971540 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7202-a85cb4" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" containerName="test-curl-7202-a85cb4" containerID="cri-o://6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba" gracePeriod=2 Nov 02 22:06:51.981090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.981068 5091 scope.go:122] "RemoveContainer" containerID="4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c" Nov 02 22:06:51.981357 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:51.981338 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c\": container with ID starting with 4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c not found: ID does not exist" containerID="4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c" Nov 02 22:06:51.981399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:51.981365 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c"} err="failed to get container status \"4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c\": rpc error: code = NotFound desc = could not find container \"4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c\": container with ID starting with 4bd0280694f227d7cbdf4e888290ddf07c4db98a892d598d271b729bf942f08c not found: ID does not exist" Nov 02 22:06:52.022809 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:52.022779 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-a85cb4\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:52.024443 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.024417 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dw7\" (UniqueName: \"kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7\") pod \"test-curl-7202-401fe2\" (UID: \"c28355f0-4117-4777-ae12-8c392dcd75e2\") " pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:52.073595 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:52.073564 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-a85cb4\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:52.080377 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.080350 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dw7\" (UniqueName: \"kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7\") pod \"test-curl-7202-401fe2\" (UID: \"c28355f0-4117-4777-ae12-8c392dcd75e2\") " pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:52.097585 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.097564 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:52.124142 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:52.124109 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-2edeb5\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:52.402774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.402738 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974"] Nov 02 22:06:52.452318 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.452282 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-ql974"] Nov 02 22:06:52.510167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.510139 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-401fe2"] Nov 02 22:06:52.510446 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:06:52.510422 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc28355f0_4117_4777_ae12_8c392dcd75e2.slice/crio-8799331dd91f59de89dceaf97e1199e16663c1627e2c6d4c981e7419a3eba6b2 WatchSource:0}: Error finding container 8799331dd91f59de89dceaf97e1199e16663c1627e2c6d4c981e7419a3eba6b2: Status 404 returned error can't find the container with id 8799331dd91f59de89dceaf97e1199e16663c1627e2c6d4c981e7419a3eba6b2 Nov 02 22:06:52.985872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.985839 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" path="/var/lib/kubelet/pods/3ff91206-fa5c-42b5-bcc2-1ccf6c754530/volumes" Nov 02 22:06:52.986437 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.986413 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88" path="/var/lib/kubelet/pods/6faf1b94-5bf3-46e6-b4ca-3d1c3ad19f88/volumes" Nov 02 22:06:52.989776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.989751 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-401fe2" event={"ID":"c28355f0-4117-4777-ae12-8c392dcd75e2","Type":"ContainerStarted","Data":"2e8c00fef18825e96eae6d5d425772ac385c4f542168a6f159b15d453dd7dc59"} Nov 02 22:06:52.989871 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:52.989783 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-401fe2" event={"ID":"c28355f0-4117-4777-ae12-8c392dcd75e2","Type":"ContainerStarted","Data":"8799331dd91f59de89dceaf97e1199e16663c1627e2c6d4c981e7419a3eba6b2"} Nov 02 22:06:53.032627 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:53.032592 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-a85cb4\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:53.083432 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:53.083402 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-2edeb5\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="3ff91206-fa5c-42b5-bcc2-1ccf6c754530" pod="opendatahub/test-curl-7202-2edeb5" Nov 02 22:06:53.197316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:53.197265 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7202-401fe2" podStartSLOduration=2.19725096 podStartE2EDuration="2.19725096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:53.194124805 +0000 UTC m=+1960.356219926" watchObservedRunningTime="2026-11-02 22:06:53.19725096 +0000 UTC m=+1960.359346079" Nov 02 22:06:54.237060 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.236997 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:54.359146 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.359115 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8\" (UniqueName: \"kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8\") pod \"9fd6f5aa-4de0-4a34-af9f-0061392f0229\" (UID: \"9fd6f5aa-4de0-4a34-af9f-0061392f0229\") " Nov 02 22:06:54.361495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.361453 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8" pod "9fd6f5aa-4de0-4a34-af9f-0061392f0229" (UID: "9fd6f5aa-4de0-4a34-af9f-0061392f0229"). InnerVolumeSpecName "kube-api-access-rwfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:54.460974 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.460942 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfb8\" (UniqueName: \"kubernetes.io/projected/9fd6f5aa-4de0-4a34-af9f-0061392f0229-kube-api-access-rwfb8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:54.648835 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.648797 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7202-401fe2"] Nov 02 22:06:54.654569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.654510 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7202-401fe2"] Nov 02 22:06:54.981253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:54.981167 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" path="/var/lib/kubelet/pods/9fd6f5aa-4de0-4a34-af9f-0061392f0229/volumes" Nov 02 22:06:55.005284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:55.005252 5091 generic.go:356] "Generic (PLEG): container finished" podID="9fd6f5aa-4de0-4a34-af9f-0061392f0229" containerID="6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba" exitCode=137 Nov 02 22:06:55.005452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:55.005312 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-a85cb4" Nov 02 22:06:55.005452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:55.005337 5091 scope.go:122] "RemoveContainer" containerID="6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba" Nov 02 22:06:55.005452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:55.005427 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7202-401fe2" podUID="c28355f0-4117-4777-ae12-8c392dcd75e2" containerName="test-curl-7202-401fe2" containerID="cri-o://2e8c00fef18825e96eae6d5d425772ac385c4f542168a6f159b15d453dd7dc59" gracePeriod=2 Nov 02 22:06:55.020710 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:55.018765 5091 scope.go:122] "RemoveContainer" containerID="6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba" Nov 02 22:06:55.021713 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:55.021374 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba\": container with ID starting with 6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba not found: ID does not exist" containerID="6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba" Nov 02 22:06:55.021713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:55.021412 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba"} err="failed to get container status \"6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba\": rpc error: code = NotFound desc = could not find container \"6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba\": container with ID starting with 6b4687a2d24ca7d6a4979109783f682b20cfeab19dd077b2de514c12703275ba not found: ID does not exist" Nov 02 22:06:57.148272 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:57.148238 5091 generic.go:356] "Generic (PLEG): container finished" podID="c28355f0-4117-4777-ae12-8c392dcd75e2" containerID="2e8c00fef18825e96eae6d5d425772ac385c4f542168a6f159b15d453dd7dc59" exitCode=137 Nov 02 22:06:57.285954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:57.285740 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:57.412739 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:57.412705 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7\" (UniqueName: \"kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7\") pod \"c28355f0-4117-4777-ae12-8c392dcd75e2\" (UID: \"c28355f0-4117-4777-ae12-8c392dcd75e2\") " Nov 02 22:06:57.414918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:57.414885 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7" pod "c28355f0-4117-4777-ae12-8c392dcd75e2" (UID: "c28355f0-4117-4777-ae12-8c392dcd75e2"). InnerVolumeSpecName "kube-api-access-s4dw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:57.523728 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:57.514779 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4dw7\" (UniqueName: \"kubernetes.io/projected/c28355f0-4117-4777-ae12-8c392dcd75e2-kube-api-access-s4dw7\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:06:58.159934 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:58.159901 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:58.160428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:58.159901 5091 scope.go:122] "RemoveContainer" containerID="2e8c00fef18825e96eae6d5d425772ac385c4f542168a6f159b15d453dd7dc59" Nov 02 22:06:58.220028 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:06:58.219989 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-401fe2\" is forbidden: User \"system:node:ip-10-0-128-36.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-36.us-west-1.compute.internal' and this object" podUID="c28355f0-4117-4777-ae12-8c392dcd75e2" pod="opendatahub/test-curl-7202-401fe2" Nov 02 22:06:58.979609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:06:58.979576 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28355f0-4117-4777-ae12-8c392dcd75e2" path="/var/lib/kubelet/pods/c28355f0-4117-4777-ae12-8c392dcd75e2/volumes" Nov 02 22:07:00.819947 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:00.819913 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8"] Nov 02 22:07:00.820363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:00.820116 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" containerID="cri-o://b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834" gracePeriod=30 Nov 02 22:07:04.155529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:04.155472 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:07:04.156389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:04.156350 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:07:06.099023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.098963 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 22:07:06.150450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150420 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.150625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150526 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.150625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150553 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.150625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150579 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.150818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150648 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.150818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150720 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.150818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150770 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:06.150818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150785 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.151021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150847 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.151021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150881 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket\") pod \"28033c5c-b32b-4d67-8326-bf25ca1afee4\" (UID: \"28033c5c-b32b-4d67-8326-bf25ca1afee4\") " Nov 02 22:07:06.151021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.150980 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:06.151215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151191 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:06.151275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151206 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:06.151375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151341 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:06.151814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151790 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.151931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151819 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.151931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151833 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28033c5c-b32b-4d67-8326-bf25ca1afee4-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.151931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151846 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.151931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.151860 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.152984 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.152952 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:06.153085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.153027 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "kube-api-access-gqmgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:06.153256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.153235 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:06.153342 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.153307 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token" pod "28033c5c-b32b-4d67-8326-bf25ca1afee4" (UID: "28033c5c-b32b-4d67-8326-bf25ca1afee4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:06.244649 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.244615 5091 generic.go:356] "Generic (PLEG): container finished" podID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerID="b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834" exitCode=0 Nov 02 22:07:06.244824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.244706 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" event={"ID":"28033c5c-b32b-4d67-8326-bf25ca1afee4","Type":"ContainerDied","Data":"b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834"} Nov 02 22:07:06.244824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.244747 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" Nov 02 22:07:06.244824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.244797 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8" event={"ID":"28033c5c-b32b-4d67-8326-bf25ca1afee4","Type":"ContainerDied","Data":"cef51de9f06232b2e7f69e719dc383346173be45ce435254c2870199be93c305"} Nov 02 22:07:06.244971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.244832 5091 scope.go:122] "RemoveContainer" containerID="b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834" Nov 02 22:07:06.253419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.253389 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.253534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.253424 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.253534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.253439 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28033c5c-b32b-4d67-8326-bf25ca1afee4-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.253534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.253455 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmgx\" (UniqueName: \"kubernetes.io/projected/28033c5c-b32b-4d67-8326-bf25ca1afee4-kube-api-access-gqmgx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:06.257448 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.257423 5091 scope.go:122] "RemoveContainer" containerID="b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834" Nov 02 22:07:06.257794 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:07:06.257773 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834\": container with ID starting with b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834 not found: ID does not exist" containerID="b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834" Nov 02 22:07:06.257873 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.257806 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834"} err="failed to get container status \"b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834\": rpc error: code = NotFound desc = could not find container \"b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834\": container with ID starting with b703d6a5261d152678607262a1a5f92c68e310d4761769358605dfb7dd88b834 not found: ID does not exist" Nov 02 22:07:06.412227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.412193 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8"] Nov 02 22:07:06.415215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.415192 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-psqw8"] Nov 02 22:07:06.978353 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:06.978320 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" path="/var/lib/kubelet/pods/28033c5c-b32b-4d67-8326-bf25ca1afee4/volumes" Nov 02 22:07:07.113725 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:07:07.113661 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:07:11.146245 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.146211 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8"] Nov 02 22:07:11.147115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.147091 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c28355f0-4117-4777-ae12-8c392dcd75e2" containerName="test-curl-7202-401fe2" Nov 02 22:07:11.147219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.147115 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28355f0-4117-4777-ae12-8c392dcd75e2" containerName="test-curl-7202-401fe2" Nov 02 22:07:11.147219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.147138 5091 cpu_manager.go:414] "containerMap: removing container" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" Nov 02 22:07:11.147219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.147147 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" Nov 02 22:07:11.147380 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.147365 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="28033c5c-b32b-4d67-8326-bf25ca1afee4" containerName="istio-proxy" Nov 02 22:07:11.147442 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.147398 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28355f0-4117-4777-ae12-8c392dcd75e2" containerName="test-curl-7202-401fe2" Nov 02 22:07:11.152465 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.152444 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.207636 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.207600 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8"] Nov 02 22:07:11.251913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.251880 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:07:11.251913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.251890 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:07:11.252101 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.251913 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:07:11.253792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.253767 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:07:11.274152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.274124 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gmk\" (UniqueName: \"kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.274273 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.274165 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.274273 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.274186 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.375056 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.375025 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gmk\" (UniqueName: \"kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.375202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.375065 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.375202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.375082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.375792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.375769 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.377496 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.377476 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.431290 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.431232 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gmk\" (UniqueName: \"kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.466376 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.466350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:11.841665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:11.841626 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8"] Nov 02 22:07:11.842883 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:07:11.842855 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7303ebd5_c8b0_4831_828c_18c79c0ed25e.slice/crio-3c51ba6770a2aa250decd4bb768a6d31a05c68f9d5b8e6faba5ce4c935c57b37 WatchSource:0}: Error finding container 3c51ba6770a2aa250decd4bb768a6d31a05c68f9d5b8e6faba5ce4c935c57b37: Status 404 returned error can't find the container with id 3c51ba6770a2aa250decd4bb768a6d31a05c68f9d5b8e6faba5ce4c935c57b37 Nov 02 22:07:12.023968 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.023934 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv"] Nov 02 22:07:12.029477 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.029447 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.157694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.128648 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2050f3-openshift-default-dockercfg-5hs9z\"" Nov 02 22:07:12.190760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.186712 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv"] Nov 02 22:07:12.353606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.353567 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drcg8\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.353875 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.353852 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354037 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354020 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354150 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354321 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354520 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354664 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354795 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.354925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.354912 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.469712 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.469788 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.469833 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.469876 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.469994 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drcg8\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.470044 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.470090 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.470133 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.470234 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.470768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.471086 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.471331 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.472162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.471649 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.481897 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.481866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.484584 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.484557 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.484792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.484760 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" event={"ID":"7303ebd5-c8b0-4831-828c-18c79c0ed25e","Type":"ContainerStarted","Data":"0f2fe8fa2e9f498112a819e122461b4f3581a27e4718e033941e4451c61b47f4"} Nov 02 22:07:12.484876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.484824 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" event={"ID":"7303ebd5-c8b0-4831-828c-18c79c0ed25e","Type":"ContainerStarted","Data":"3c51ba6770a2aa250decd4bb768a6d31a05c68f9d5b8e6faba5ce4c935c57b37"} Nov 02 22:07:12.492247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.492224 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.527180 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.527123 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.527428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.527409 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drcg8\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8\") pod \"e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:12.598954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.598895 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" podStartSLOduration=1.598880545 podStartE2EDuration="1.598880545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:12.59653315 +0000 UTC m=+1979.758628293" watchObservedRunningTime="2026-11-02 22:07:12.598880545 +0000 UTC m=+1979.760975665" Nov 02 22:07:12.652195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:12.652163 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:13.138715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.138640 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv"] Nov 02 22:07:13.139232 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:07:13.139200 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae31de7_5263_4371_b452_ebb1b1edc4de.slice/crio-8e117cf901e47893a357ea4ce31413ba74966fd4e08ed39c833e6b7d718121f5 WatchSource:0}: Error finding container 8e117cf901e47893a357ea4ce31413ba74966fd4e08ed39c833e6b7d718121f5: Status 404 returned error can't find the container with id 8e117cf901e47893a357ea4ce31413ba74966fd4e08ed39c833e6b7d718121f5 Nov 02 22:07:13.142473 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.142434 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:07:13.142581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.142518 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:07:13.142581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.142561 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:07:13.497239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.497195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" event={"ID":"2ae31de7-5263-4371-b452-ebb1b1edc4de","Type":"ContainerStarted","Data":"20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f"} Nov 02 22:07:13.497780 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.497269 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" event={"ID":"2ae31de7-5263-4371-b452-ebb1b1edc4de","Type":"ContainerStarted","Data":"8e117cf901e47893a357ea4ce31413ba74966fd4e08ed39c833e6b7d718121f5"} Nov 02 22:07:13.499254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.499228 5091 generic.go:356] "Generic (PLEG): container finished" podID="7303ebd5-c8b0-4831-828c-18c79c0ed25e" containerID="0f2fe8fa2e9f498112a819e122461b4f3581a27e4718e033941e4451c61b47f4" exitCode=0 Nov 02 22:07:13.499381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.499272 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" event={"ID":"7303ebd5-c8b0-4831-828c-18c79c0ed25e","Type":"ContainerDied","Data":"0f2fe8fa2e9f498112a819e122461b4f3581a27e4718e033941e4451c61b47f4"} Nov 02 22:07:13.614720 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.614632 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podStartSLOduration=2.614618599 podStartE2EDuration="2.614618599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:13.612392334 +0000 UTC m=+1980.774487453" watchObservedRunningTime="2026-11-02 22:07:13.614618599 +0000 UTC m=+1980.776713790" Nov 02 22:07:13.652614 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:13.652524 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:14.658605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.658581 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:14.687971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.687940 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:14.723822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.723797 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca\") pod \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " Nov 02 22:07:14.724013 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.723998 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk\" (UniqueName: \"kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk\") pod \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " Nov 02 22:07:14.724100 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.724035 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca\") pod \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\" (UID: \"7303ebd5-c8b0-4831-828c-18c79c0ed25e\") " Nov 02 22:07:14.724484 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.724453 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca" pod "7303ebd5-c8b0-4831-828c-18c79c0ed25e" (UID: "7303ebd5-c8b0-4831-828c-18c79c0ed25e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:14.724699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.724682 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:14.726030 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.726002 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca" pod "7303ebd5-c8b0-4831-828c-18c79c0ed25e" (UID: "7303ebd5-c8b0-4831-828c-18c79c0ed25e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:07:14.726180 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.726161 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk" pod "7303ebd5-c8b0-4831-828c-18c79c0ed25e" (UID: "7303ebd5-c8b0-4831-828c-18c79c0ed25e"). InnerVolumeSpecName "kube-api-access-n5gmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:14.825968 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.825906 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303ebd5-c8b0-4831-828c-18c79c0ed25e-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:14.825968 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:14.825934 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gmk\" (UniqueName: \"kubernetes.io/projected/7303ebd5-c8b0-4831-828c-18c79c0ed25e-kube-api-access-n5gmk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:15.515451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:15.515422 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" Nov 02 22:07:15.515639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:15.515464 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8" event={"ID":"7303ebd5-c8b0-4831-828c-18c79c0ed25e","Type":"ContainerDied","Data":"3c51ba6770a2aa250decd4bb768a6d31a05c68f9d5b8e6faba5ce4c935c57b37"} Nov 02 22:07:15.515639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:15.515519 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c51ba6770a2aa250decd4bb768a6d31a05c68f9d5b8e6faba5ce4c935c57b37" Nov 02 22:07:15.515804 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:15.515785 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:15.516881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:15.516860 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:07:19.952589 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:19.952555 5091 scope.go:122] "RemoveContainer" containerID="45d7deed6a62634da1b42d8f35d9dcb9ca253fdc5f241bdcf978576e911aa417" Nov 02 22:07:19.965728 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:19.965698 5091 scope.go:122] "RemoveContainer" containerID="8a35beb5a090b7206e14ed9c79a30064fdb4e76bc27d98da75e5be825ca55cf3" Nov 02 22:07:19.978434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:19.978408 5091 scope.go:122] "RemoveContainer" containerID="8c6f726d6997757beaf77233e5c9ac7131e8daecb9083904dac90104c8ae4a3c" Nov 02 22:07:19.993201 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:19.992423 5091 scope.go:122] "RemoveContainer" containerID="c832c9d30fda95fe29aa2abfb1fb6f45bfd35f2bd612e318d1fe29f8162b5f9a" Nov 02 22:07:37.116750 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:07:37.116706 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:07:45.197534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.197502 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8"] Nov 02 22:07:45.198184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.198169 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7303ebd5-c8b0-4831-828c-18c79c0ed25e" containerName="revoke-keys" Nov 02 22:07:45.198238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.198185 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7303ebd5-c8b0-4831-828c-18c79c0ed25e" containerName="revoke-keys" Nov 02 22:07:45.198316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.198307 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7303ebd5-c8b0-4831-828c-18c79c0ed25e" containerName="revoke-keys" Nov 02 22:07:45.202038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.202020 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.259048 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.259017 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8"] Nov 02 22:07:45.290183 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.290152 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.290355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.290313 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzpc\" (UniqueName: \"kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.290487 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.290465 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.302640 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.302617 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:07:45.302640 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.302622 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:07:45.302807 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.302656 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:07:45.303760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.303745 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:07:45.391772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.391742 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.391917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.391844 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzpc\" (UniqueName: \"kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.391971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.391941 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.392570 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.392546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.394099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.394082 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.447898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.447832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzpc\" (UniqueName: \"kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc\") pod \"maas-api-revoke-keys-e2e-api-a-224f53-lnxb8\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.515838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.515807 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:45.679611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.679585 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc"] Nov 02 22:07:45.679819 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.679797 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" containerID="cri-o://0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de" gracePeriod=30 Nov 02 22:07:45.683744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.683719 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv"] Nov 02 22:07:45.683915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.683896 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" containerID="cri-o://7d89cc3448e6ce41cecbf1df20cdfeb09f5006fcda1f0937699d8011c9e398cc" gracePeriod=30 Nov 02 22:07:45.687185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.687161 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch"] Nov 02 22:07:45.687319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.687305 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" containerID="cri-o://c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1" gracePeriod=30 Nov 02 22:07:45.812081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.811998 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h"] Nov 02 22:07:45.812325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.812255 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" containerID="cri-o://09010a4935ae2a047ae799041898b1d64f20d6f169af077142d32d77a5a4fc84" gracePeriod=30 Nov 02 22:07:45.817454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.816468 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n"] Nov 02 22:07:45.817454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.816723 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" containerID="cri-o://4bc0eb5b8d0b96e9d9548e876191d59be919489c58057330c6394a08077d3dea" gracePeriod=30 Nov 02 22:07:45.821592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.821567 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p"] Nov 02 22:07:45.821778 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.821758 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" containerID="cri-o://bee3a8782acff51770d8322b14332e2ee0ecf1a3969873d3c527ed686f17e471" gracePeriod=30 Nov 02 22:07:45.912203 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:07:45.912174 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4d0f5f_17b5_46ec_b669_201516c5c6c2.slice/crio-9a78bc6e61f79860c5afd8865a2e680120535ed35be61c1f1b7e4ee51222e327 WatchSource:0}: Error finding container 9a78bc6e61f79860c5afd8865a2e680120535ed35be61c1f1b7e4ee51222e327: Status 404 returned error can't find the container with id 9a78bc6e61f79860c5afd8865a2e680120535ed35be61c1f1b7e4ee51222e327 Nov 02 22:07:45.912542 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.912516 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8"] Nov 02 22:07:45.940209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:45.938926 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" event={"ID":"0e4d0f5f-17b5-46ec-b669-201516c5c6c2","Type":"ContainerStarted","Data":"9a78bc6e61f79860c5afd8865a2e680120535ed35be61c1f1b7e4ee51222e327"} Nov 02 22:07:47.049711 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:47.048703 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" event={"ID":"0e4d0f5f-17b5-46ec-b669-201516c5c6c2","Type":"ContainerStarted","Data":"0cdfb0defe8627f5f413de6a777adbf3aa1785df2a9a0c886a1f42bf2e3478eb"} Nov 02 22:07:47.163629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:47.163574 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" podStartSLOduration=2.163560472 podStartE2EDuration="2.163560472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:47.161091742 +0000 UTC m=+2014.323186861" watchObservedRunningTime="2026-11-02 22:07:47.163560472 +0000 UTC m=+2014.325655613" Nov 02 22:07:48.057647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:48.057613 5091 generic.go:356] "Generic (PLEG): container finished" podID="0e4d0f5f-17b5-46ec-b669-201516c5c6c2" containerID="0cdfb0defe8627f5f413de6a777adbf3aa1785df2a9a0c886a1f42bf2e3478eb" exitCode=0 Nov 02 22:07:48.058057 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:48.057696 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" event={"ID":"0e4d0f5f-17b5-46ec-b669-201516c5c6c2","Type":"ContainerDied","Data":"0cdfb0defe8627f5f413de6a777adbf3aa1785df2a9a0c886a1f42bf2e3478eb"} Nov 02 22:07:49.227768 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.227741 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:49.277407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.277379 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc\" (UniqueName: \"kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc\") pod \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " Nov 02 22:07:49.277551 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.277481 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca\") pod \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " Nov 02 22:07:49.277551 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.277534 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca\") pod \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\" (UID: \"0e4d0f5f-17b5-46ec-b669-201516c5c6c2\") " Nov 02 22:07:49.277866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.277837 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca" pod "0e4d0f5f-17b5-46ec-b669-201516c5c6c2" (UID: "0e4d0f5f-17b5-46ec-b669-201516c5c6c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:49.278064 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.278050 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:49.279645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.279618 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca" pod "0e4d0f5f-17b5-46ec-b669-201516c5c6c2" (UID: "0e4d0f5f-17b5-46ec-b669-201516c5c6c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:07:49.279645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.279628 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc" pod "0e4d0f5f-17b5-46ec-b669-201516c5c6c2" (UID: "0e4d0f5f-17b5-46ec-b669-201516c5c6c2"). InnerVolumeSpecName "kube-api-access-dgzpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:49.380058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.380025 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzpc\" (UniqueName: \"kubernetes.io/projected/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-kube-api-access-dgzpc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:49.380216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:49.380063 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4d0f5f-17b5-46ec-b669-201516c5c6c2-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:50.073709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:50.073593 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" Nov 02 22:07:50.073709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:50.073622 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8" event={"ID":"0e4d0f5f-17b5-46ec-b669-201516c5c6c2","Type":"ContainerDied","Data":"9a78bc6e61f79860c5afd8865a2e680120535ed35be61c1f1b7e4ee51222e327"} Nov 02 22:07:50.073709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:50.073684 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a78bc6e61f79860c5afd8865a2e680120535ed35be61c1f1b7e4ee51222e327" Nov 02 22:07:50.209273 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:50.209234 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:07:50.209432 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:50.209297 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:07:51.077089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.076868 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 22:07:51.080359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.080332 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 22:07:51.089129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.088831 5091 generic.go:356] "Generic (PLEG): container finished" podID="515c9af2-cd82-407e-947c-10cf076651d4" containerID="09010a4935ae2a047ae799041898b1d64f20d6f169af077142d32d77a5a4fc84" exitCode=0 Nov 02 22:07:51.089129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.089063 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" event={"ID":"515c9af2-cd82-407e-947c-10cf076651d4","Type":"ContainerDied","Data":"09010a4935ae2a047ae799041898b1d64f20d6f169af077142d32d77a5a4fc84"} Nov 02 22:07:51.092381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.092200 5091 generic.go:356] "Generic (PLEG): container finished" podID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerID="7d89cc3448e6ce41cecbf1df20cdfeb09f5006fcda1f0937699d8011c9e398cc" exitCode=0 Nov 02 22:07:51.092381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.092318 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" event={"ID":"592d0e95-5afb-4f73-8c3e-6594e08388c2","Type":"ContainerDied","Data":"7d89cc3448e6ce41cecbf1df20cdfeb09f5006fcda1f0937699d8011c9e398cc"} Nov 02 22:07:51.094996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.094974 5091 generic.go:356] "Generic (PLEG): container finished" podID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerID="4bc0eb5b8d0b96e9d9548e876191d59be919489c58057330c6394a08077d3dea" exitCode=0 Nov 02 22:07:51.095118 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.095101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" event={"ID":"0709d9c9-5704-435e-909a-ad40dfdb78e0","Type":"ContainerDied","Data":"4bc0eb5b8d0b96e9d9548e876191d59be919489c58057330c6394a08077d3dea"} Nov 02 22:07:51.096533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.096514 5091 generic.go:356] "Generic (PLEG): container finished" podID="411e3bef-199a-4a56-8e53-7477d83be204" containerID="c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1" exitCode=0 Nov 02 22:07:51.096639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.096616 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" Nov 02 22:07:51.096850 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.096834 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" event={"ID":"411e3bef-199a-4a56-8e53-7477d83be204","Type":"ContainerDied","Data":"c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1"} Nov 02 22:07:51.096931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.096910 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch" event={"ID":"411e3bef-199a-4a56-8e53-7477d83be204","Type":"ContainerDied","Data":"7b0cd3f411e1ef298b547bcc46c57521e5469d391915edfc2a39457c6ca5a799"} Nov 02 22:07:51.096992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.096949 5091 scope.go:122] "RemoveContainer" containerID="c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1" Nov 02 22:07:51.099014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.098992 5091 generic.go:356] "Generic (PLEG): container finished" podID="843bd159-e589-425e-b160-4d5e6f28e638" containerID="0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de" exitCode=0 Nov 02 22:07:51.099167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.099046 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" Nov 02 22:07:51.099167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.099074 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" event={"ID":"843bd159-e589-425e-b160-4d5e6f28e638","Type":"ContainerDied","Data":"0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de"} Nov 02 22:07:51.099167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.099109 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc" event={"ID":"843bd159-e589-425e-b160-4d5e6f28e638","Type":"ContainerDied","Data":"52778e03e35f64b28148e483b5575a27920babb5b7ae7a28a17abf15c0e9b7c5"} Nov 02 22:07:51.101061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.101040 5091 generic.go:356] "Generic (PLEG): container finished" podID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerID="bee3a8782acff51770d8322b14332e2ee0ecf1a3969873d3c527ed686f17e471" exitCode=0 Nov 02 22:07:51.101228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.101213 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" event={"ID":"857f1f53-f73a-4026-9c6c-5a9013252e88","Type":"ContainerDied","Data":"bee3a8782acff51770d8322b14332e2ee0ecf1a3969873d3c527ed686f17e471"} Nov 02 22:07:51.108620 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.108591 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.108891 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.108869 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.109040 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.109021 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.109164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.109149 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.109292 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.109275 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.109472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.109446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.109650 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.109632 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.109870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.109851 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.110061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110044 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.110181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110167 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.110289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110275 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.110423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110407 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.110546 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110532 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.110730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110711 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.110918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.110902 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.111050 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.111036 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.111212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.111196 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token\") pod \"411e3bef-199a-4a56-8e53-7477d83be204\" (UID: \"411e3bef-199a-4a56-8e53-7477d83be204\") " Nov 02 22:07:51.111348 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.111334 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket\") pod \"843bd159-e589-425e-b160-4d5e6f28e638\" (UID: \"843bd159-e589-425e-b160-4d5e6f28e638\") " Nov 02 22:07:51.112449 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.112426 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.112912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.112890 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.113398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.113338 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.114247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.113542 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.117181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.115169 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.117181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.115497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.117181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.116425 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.117181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.116629 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.117181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.116939 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.118820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.118372 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "kube-api-access-qlrjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.119200 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.119181 5091 scope.go:122] "RemoveContainer" containerID="c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1" Nov 02 22:07:51.120288 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.119770 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "kube-api-access-26vfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.120288 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.120022 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.121062 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.121026 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.122259 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.122141 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.123315 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.123281 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.124065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.123767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.124237 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.124059 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token" pod "843bd159-e589-425e-b160-4d5e6f28e638" (UID: "843bd159-e589-425e-b160-4d5e6f28e638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.124292 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:07:51.124257 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1\": container with ID starting with c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1 not found: ID does not exist" containerID="c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1" Nov 02 22:07:51.124381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.124296 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1"} err="failed to get container status \"c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1\": rpc error: code = NotFound desc = could not find container \"c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1\": container with ID starting with c786226b866346aa9445dec25ddc970966106b415eed91d1c7030d5b620f0cc1 not found: ID does not exist" Nov 02 22:07:51.124381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.124325 5091 scope.go:122] "RemoveContainer" containerID="0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de" Nov 02 22:07:51.127178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.126298 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo" pod "411e3bef-199a-4a56-8e53-7477d83be204" (UID: "411e3bef-199a-4a56-8e53-7477d83be204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.128655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.128633 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 22:07:51.134712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.134623 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 22:07:51.138590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.138106 5091 scope.go:122] "RemoveContainer" containerID="0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de" Nov 02 22:07:51.139074 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:07:51.138954 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de\": container with ID starting with 0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de not found: ID does not exist" containerID="0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de" Nov 02 22:07:51.139074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.138983 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de"} err="failed to get container status \"0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de\": rpc error: code = NotFound desc = could not find container \"0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de\": container with ID starting with 0856c00d91df3ef9a1d41bdd816472dbcb8742557ca06d59bec4e05bbadf57de not found: ID does not exist" Nov 02 22:07:51.141984 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.141964 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 22:07:51.212952 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.212926 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.213159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.212970 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.213159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213001 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.213159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213031 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.213159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213120 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.213159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213158 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213202 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213233 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213284 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213325 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213378 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213438 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213477 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.213588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213563 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213651 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213716 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213760 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213800 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213857 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213893 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data\") pod \"515c9af2-cd82-407e-947c-10cf076651d4\" (UID: \"515c9af2-cd82-407e-947c-10cf076651d4\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.213981 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.214014 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.214055 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.214102 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.214145 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6\") pod \"592d0e95-5afb-4f73-8c3e-6594e08388c2\" (UID: \"592d0e95-5afb-4f73-8c3e-6594e08388c2\") " Nov 02 22:07:51.214261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.214183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data\") pod \"0709d9c9-5704-435e-909a-ad40dfdb78e0\" (UID: \"0709d9c9-5704-435e-909a-ad40dfdb78e0\") " Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.214993 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215018 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215034 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215050 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/411e3bef-199a-4a56-8e53-7477d83be204-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215064 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215079 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215093 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26vfc\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-kube-api-access-26vfc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215108 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215122 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/843bd159-e589-425e-b160-4d5e6f28e638-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215137 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215150 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215166 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/411e3bef-199a-4a56-8e53-7477d83be204-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215180 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215194 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/411e3bef-199a-4a56-8e53-7477d83be204-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215210 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/411e3bef-199a-4a56-8e53-7477d83be204-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215225 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/843bd159-e589-425e-b160-4d5e6f28e638-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215240 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlrjm\" (UniqueName: \"kubernetes.io/projected/843bd159-e589-425e-b160-4d5e6f28e638-kube-api-access-qlrjm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215257 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/843bd159-e589-425e-b160-4d5e6f28e638-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215346 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215529 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215684 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.215986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.215889 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.217810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.217014 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.217810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.217559 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.217989 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.217841 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.218239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.218209 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.219850 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.219147 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.219850 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.219164 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.219850 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.219405 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.219850 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.219603 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.220125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.220070 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.220196 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.220128 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.220448 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.220424 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224119 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224176 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224258 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224290 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224417 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "kube-api-access-dl4h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224433 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224587 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb" pod "0709d9c9-5704-435e-909a-ad40dfdb78e0" (UID: "0709d9c9-5704-435e-909a-ad40dfdb78e0"). InnerVolumeSpecName "kube-api-access-xn6fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224803 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.224871 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.225033 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo" pod "592d0e95-5afb-4f73-8c3e-6594e08388c2" (UID: "592d0e95-5afb-4f73-8c3e-6594e08388c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.225379 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "kube-api-access-whtkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.226002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.225942 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy" pod "515c9af2-cd82-407e-947c-10cf076651d4" (UID: "515c9af2-cd82-407e-947c-10cf076651d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.316016 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.315981 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316188 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316044 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316188 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316097 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316188 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316136 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316206 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316280 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316308 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.316371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316317 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316384 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316488 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.316645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316545 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket\") pod \"857f1f53-f73a-4026-9c6c-5a9013252e88\" (UID: \"857f1f53-f73a-4026-9c6c-5a9013252e88\") " Nov 02 22:07:51.316645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.316923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316894 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.316923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.316911 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.317358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317332 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317365 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317381 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtkj\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-kube-api-access-whtkj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317396 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317410 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn6fb\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-kube-api-access-xn6fb\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317423 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317437 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317450 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317471 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317485 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317499 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592d0e95-5afb-4f73-8c3e-6594e08388c2-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317513 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317528 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4h6\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-kube-api-access-dl4h6\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317541 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317555 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317568 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317581 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515c9af2-cd82-407e-947c-10cf076651d4-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317594 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0709d9c9-5704-435e-909a-ad40dfdb78e0-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317608 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317622 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317636 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317651 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317665 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317695 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0709d9c9-5704-435e-909a-ad40dfdb78e0-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317709 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317722 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515c9af2-cd82-407e-947c-10cf076651d4-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317737 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317753 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592d0e95-5afb-4f73-8c3e-6594e08388c2-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317767 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515c9af2-cd82-407e-947c-10cf076651d4-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317779 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515c9af2-cd82-407e-947c-10cf076651d4-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317794 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.317820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.317809 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1f53-f73a-4026-9c6c-5a9013252e88-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.318971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.318586 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.318971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.318720 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "kube-api-access-zpk96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.318971 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.318767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.319397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.319380 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy" pod "857f1f53-f73a-4026-9c6c-5a9013252e88" (UID: "857f1f53-f73a-4026-9c6c-5a9013252e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.418924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.418898 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.418924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.418924 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.419074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.418934 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1f53-f73a-4026-9c6c-5a9013252e88-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.419074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:51.418944 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpk96\" (UniqueName: \"kubernetes.io/projected/857f1f53-f73a-4026-9c6c-5a9013252e88-kube-api-access-zpk96\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:52.008539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.008417 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch"] Nov 02 22:07:52.013584 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.013554 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-49wch"] Nov 02 22:07:52.111972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.111941 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc"] Nov 02 22:07:52.112741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.112699 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" event={"ID":"857f1f53-f73a-4026-9c6c-5a9013252e88","Type":"ContainerDied","Data":"7da5ec6302f8c96f06bfeb98991cfd8ffe815577e2d104bbd40816a5b15edefc"} Nov 02 22:07:52.112902 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.112725 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p" Nov 02 22:07:52.112902 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.112780 5091 scope.go:122] "RemoveContainer" containerID="bee3a8782acff51770d8322b14332e2ee0ecf1a3969873d3c527ed686f17e471" Nov 02 22:07:52.114358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.114269 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" Nov 02 22:07:52.114358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.114265 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h" event={"ID":"515c9af2-cd82-407e-947c-10cf076651d4","Type":"ContainerDied","Data":"c297dc249d889d26cba3307e0ca079e8c8e9c200d6d132492fb2f94daf33545e"} Nov 02 22:07:52.119164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.115870 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" Nov 02 22:07:52.119164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.115873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv" event={"ID":"592d0e95-5afb-4f73-8c3e-6594e08388c2","Type":"ContainerDied","Data":"7a2444a404d08fb108e92bd4f881bce519651d1f003d5eaa3859ec5fc401bb6c"} Nov 02 22:07:52.119164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.117089 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-k2znc"] Nov 02 22:07:52.119164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.117393 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" Nov 02 22:07:52.119164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.117413 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n" event={"ID":"0709d9c9-5704-435e-909a-ad40dfdb78e0","Type":"ContainerDied","Data":"bfb5ddfe817ed4ff9063d2387e5e147c816cfa19d3abe02654ba9a1d54c7896d"} Nov 02 22:07:52.126972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.126955 5091 scope.go:122] "RemoveContainer" containerID="09010a4935ae2a047ae799041898b1d64f20d6f169af077142d32d77a5a4fc84" Nov 02 22:07:52.138692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.138663 5091 scope.go:122] "RemoveContainer" containerID="7d89cc3448e6ce41cecbf1df20cdfeb09f5006fcda1f0937699d8011c9e398cc" Nov 02 22:07:52.148896 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.148881 5091 scope.go:122] "RemoveContainer" containerID="4bc0eb5b8d0b96e9d9548e876191d59be919489c58057330c6394a08077d3dea" Nov 02 22:07:52.278497 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.278421 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv"] Nov 02 22:07:52.283213 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.283186 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-zt4wv"] Nov 02 22:07:52.391929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.391902 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h"] Nov 02 22:07:52.395381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.395361 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-56k2h"] Nov 02 22:07:52.502434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.502402 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n"] Nov 02 22:07:52.505906 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.505882 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-qb48n"] Nov 02 22:07:52.615012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.614916 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p"] Nov 02 22:07:52.618494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.618469 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-llz7p"] Nov 02 22:07:52.978051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.978019 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" path="/var/lib/kubelet/pods/0709d9c9-5704-435e-909a-ad40dfdb78e0/volumes" Nov 02 22:07:52.978543 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.978527 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411e3bef-199a-4a56-8e53-7477d83be204" path="/var/lib/kubelet/pods/411e3bef-199a-4a56-8e53-7477d83be204/volumes" Nov 02 22:07:52.979025 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.979011 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515c9af2-cd82-407e-947c-10cf076651d4" path="/var/lib/kubelet/pods/515c9af2-cd82-407e-947c-10cf076651d4/volumes" Nov 02 22:07:52.979800 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.979783 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" path="/var/lib/kubelet/pods/592d0e95-5afb-4f73-8c3e-6594e08388c2/volumes" Nov 02 22:07:52.980251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.980238 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843bd159-e589-425e-b160-4d5e6f28e638" path="/var/lib/kubelet/pods/843bd159-e589-425e-b160-4d5e6f28e638/volumes" Nov 02 22:07:52.980660 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:52.980647 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" path="/var/lib/kubelet/pods/857f1f53-f73a-4026-9c6c-5a9013252e88/volumes" Nov 02 22:07:57.604881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.604806 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh"] Nov 02 22:07:57.605412 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605398 5091 cpu_manager.go:414] "containerMap: removing container" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" Nov 02 22:07:57.605472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605412 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" Nov 02 22:07:57.605472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605435 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" Nov 02 22:07:57.605472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605440 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" Nov 02 22:07:57.605472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605449 5091 cpu_manager.go:414] "containerMap: removing container" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" Nov 02 22:07:57.605472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605455 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605481 5091 cpu_manager.go:414] "containerMap: removing container" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605487 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605499 5091 cpu_manager.go:414] "containerMap: removing container" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605504 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605511 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0e4d0f5f-17b5-46ec-b669-201516c5c6c2" containerName="revoke-keys" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605516 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4d0f5f-17b5-46ec-b669-201516c5c6c2" containerName="revoke-keys" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605527 5091 cpu_manager.go:414] "containerMap: removing container" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" Nov 02 22:07:57.605619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605532 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605647 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="843bd159-e589-425e-b160-4d5e6f28e638" containerName="istio-proxy" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605657 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0709d9c9-5704-435e-909a-ad40dfdb78e0" containerName="istio-proxy" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605662 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="515c9af2-cd82-407e-947c-10cf076651d4" containerName="istio-proxy" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605688 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="592d0e95-5afb-4f73-8c3e-6594e08388c2" containerName="istio-proxy" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605699 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="411e3bef-199a-4a56-8e53-7477d83be204" containerName="istio-proxy" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605707 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4d0f5f-17b5-46ec-b669-201516c5c6c2" containerName="revoke-keys" Nov 02 22:07:57.605888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.605714 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="857f1f53-f73a-4026-9c6c-5a9013252e88" containerName="istio-proxy" Nov 02 22:07:57.611166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.611146 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.665538 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.665507 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh"] Nov 02 22:07:57.710770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.710739 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:07:57.710770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.710760 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:07:57.711092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.711065 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:07:57.712224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.712202 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:07:57.714142 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.714119 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.714248 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.714229 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5l4\" (UniqueName: \"kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.714515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.714493 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.816212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.816181 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.816370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.816237 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.816370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.816284 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5l4\" (UniqueName: \"kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.816890 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.816866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.818643 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.818623 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.872639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.872612 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5l4\" (UniqueName: \"kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4\") pod \"maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:57.925004 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:57.924977 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:07:58.304264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:58.304239 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh"] Nov 02 22:07:58.305114 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:07:58.305089 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192ff04e_2819_46fe_bc68_84c0113873df.slice/crio-e488a4c39e646af00ff032b4bd37621717ce2812f7c67e3dec96f0ef6a99d4fb WatchSource:0}: Error finding container e488a4c39e646af00ff032b4bd37621717ce2812f7c67e3dec96f0ef6a99d4fb: Status 404 returned error can't find the container with id e488a4c39e646af00ff032b4bd37621717ce2812f7c67e3dec96f0ef6a99d4fb Nov 02 22:07:59.044863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:59.044833 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt"] Nov 02 22:07:59.094082 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:59.094051 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd7c9c1-26jjt"] Nov 02 22:07:59.206482 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:59.206441 5091 generic.go:356] "Generic (PLEG): container finished" podID="192ff04e-2819-46fe-bc68-84c0113873df" containerID="48c70bc37be1c3135e3eb7e124c1166f5c4d631f71cb5a7e53020c104988ff50" exitCode=0 Nov 02 22:07:59.206637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:59.206572 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" event={"ID":"192ff04e-2819-46fe-bc68-84c0113873df","Type":"ContainerDied","Data":"48c70bc37be1c3135e3eb7e124c1166f5c4d631f71cb5a7e53020c104988ff50"} Nov 02 22:07:59.206738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:07:59.206661 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" event={"ID":"192ff04e-2819-46fe-bc68-84c0113873df","Type":"ContainerStarted","Data":"e488a4c39e646af00ff032b4bd37621717ce2812f7c67e3dec96f0ef6a99d4fb"} Nov 02 22:08:00.392295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.392272 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:08:00.562854 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.562775 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca\") pod \"192ff04e-2819-46fe-bc68-84c0113873df\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " Nov 02 22:08:00.563036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.562888 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca\") pod \"192ff04e-2819-46fe-bc68-84c0113873df\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " Nov 02 22:08:00.563104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.563051 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4\" (UniqueName: \"kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4\") pod \"192ff04e-2819-46fe-bc68-84c0113873df\" (UID: \"192ff04e-2819-46fe-bc68-84c0113873df\") " Nov 02 22:08:00.563165 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.563110 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca" pod "192ff04e-2819-46fe-bc68-84c0113873df" (UID: "192ff04e-2819-46fe-bc68-84c0113873df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:00.563482 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.563461 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:00.564983 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.564960 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca" pod "192ff04e-2819-46fe-bc68-84c0113873df" (UID: "192ff04e-2819-46fe-bc68-84c0113873df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:00.565207 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.565190 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4" pod "192ff04e-2819-46fe-bc68-84c0113873df" (UID: "192ff04e-2819-46fe-bc68-84c0113873df"). InnerVolumeSpecName "kube-api-access-ng5l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:00.664313 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.664277 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng5l4\" (UniqueName: \"kubernetes.io/projected/192ff04e-2819-46fe-bc68-84c0113873df-kube-api-access-ng5l4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:00.664313 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.664305 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192ff04e-2819-46fe-bc68-84c0113873df-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:00.674324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.674294 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj"] Nov 02 22:08:00.674479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.674459 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" containerID="cri-o://442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb" gracePeriod=30 Nov 02 22:08:00.977655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:00.977621 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3605d602-28c2-438e-9fd7-3a570386dc96" path="/var/lib/kubelet/pods/3605d602-28c2-438e-9fd7-3a570386dc96/volumes" Nov 02 22:08:01.243071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:01.242952 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" Nov 02 22:08:01.243225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:01.243076 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh" event={"ID":"192ff04e-2819-46fe-bc68-84c0113873df","Type":"ContainerDied","Data":"e488a4c39e646af00ff032b4bd37621717ce2812f7c67e3dec96f0ef6a99d4fb"} Nov 02 22:08:01.243225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:01.243121 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e488a4c39e646af00ff032b4bd37621717ce2812f7c67e3dec96f0ef6a99d4fb" Nov 02 22:08:05.657685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:05.657643 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:08:05.658057 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:05.657726 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:08:05.957856 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:05.957798 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 22:08:06.043251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043214 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043264 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043300 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043400 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043443 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043533 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043562 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043609 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043708 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:06.043810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043716 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data\") pod \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\" (UID: \"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd\") " Nov 02 22:08:06.043810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043759 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:06.043810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.043800 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:06.044091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044010 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:06.044091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044065 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:06.044808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044582 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.044808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044609 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.044808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044620 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.044808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044631 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.044808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.044640 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.045841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.045816 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:06.045958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.045933 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:06.046062 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.046044 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "kube-api-access-2k6cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:06.046257 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.046230 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo" pod "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" (UID: "5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:08:06.146076 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.146040 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6cw\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-kube-api-access-2k6cw\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.146076 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.146073 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.146076 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.146084 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.146311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.146093 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:06.281981 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.281897 5091 generic.go:356] "Generic (PLEG): container finished" podID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerID="442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb" exitCode=0 Nov 02 22:08:06.281981 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.281946 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" Nov 02 22:08:06.282166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.281982 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" event={"ID":"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd","Type":"ContainerDied","Data":"442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb"} Nov 02 22:08:06.282166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.282041 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj" event={"ID":"5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd","Type":"ContainerDied","Data":"ac851167bc85670d7064754571bd8a4ebefa3a46403e5649289eb873b2cf878f"} Nov 02 22:08:06.282166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.282070 5091 scope.go:122] "RemoveContainer" containerID="442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb" Nov 02 22:08:06.293682 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.293656 5091 scope.go:122] "RemoveContainer" containerID="442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb" Nov 02 22:08:06.293962 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:08:06.293943 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb\": container with ID starting with 442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb not found: ID does not exist" containerID="442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb" Nov 02 22:08:06.294006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.293971 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb"} err="failed to get container status \"442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb\": rpc error: code = NotFound desc = could not find container \"442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb\": container with ID starting with 442a9445404a03d3e6d0d970a949b9ca4b730fe8b0962bac35bed57019c657eb not found: ID does not exist" Nov 02 22:08:06.446659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.446614 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj"] Nov 02 22:08:06.481017 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.480988 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-gtctj"] Nov 02 22:08:06.977128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:06.977090 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" path="/var/lib/kubelet/pods/5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd/volumes" Nov 02 22:08:07.113862 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:08:07.113825 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:08:19.536318 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.536282 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9"] Nov 02 22:08:19.536899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.536885 5091 cpu_manager.go:414] "containerMap: removing container" podUID="192ff04e-2819-46fe-bc68-84c0113873df" containerName="revoke-keys" Nov 02 22:08:19.536949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.536900 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192ff04e-2819-46fe-bc68-84c0113873df" containerName="revoke-keys" Nov 02 22:08:19.536949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.536914 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" Nov 02 22:08:19.536949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.536920 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" Nov 02 22:08:19.537042 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.537025 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="192ff04e-2819-46fe-bc68-84c0113873df" containerName="revoke-keys" Nov 02 22:08:19.537042 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.537035 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0c8f89-1afe-41ad-85cf-6fbdb9c7ecbd" containerName="istio-proxy" Nov 02 22:08:19.542079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.542055 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655447 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jhs\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655624 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655748 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655785 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655848 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.655997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.655940 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.656161 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.656043 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.763016 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.702454 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9"] Nov 02 22:08:19.794797 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.723061 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17435d-openshift-default-dockercfg-2z2kc\"" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.797902 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.797953 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798001 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798066 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798132 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798173 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798196 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jhs\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.798259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.799557 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.799845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.800543 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.801716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.802257 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.802526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.819581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.803933 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.880474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.880410 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:19.885782 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:19.884592 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jhs\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs\") pod \"e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:20.164832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:20.164797 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:20.600763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:20.600731 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9"] Nov 02 22:08:20.601949 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:08:20.601918 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048fb8f1_e23d_40ae_bcd4_f873adf95ae8.slice/crio-63480dc52784ade3255f087fead2f7840ff654e64d01689d0f651a0c41c797af WatchSource:0}: Error finding container 63480dc52784ade3255f087fead2f7840ff654e64d01689d0f651a0c41c797af: Status 404 returned error can't find the container with id 63480dc52784ade3255f087fead2f7840ff654e64d01689d0f651a0c41c797af Nov 02 22:08:20.604786 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:20.604755 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:08:20.604859 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:20.604821 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:08:20.604859 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:20.604849 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:08:21.400973 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:21.400933 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" event={"ID":"048fb8f1-e23d-40ae-bcd4-f873adf95ae8","Type":"ContainerStarted","Data":"b33bb3a1baf2de332c1888ce7c3659f5e968d9a4d615c07a2447b0dcb192f915"} Nov 02 22:08:21.401167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:21.401003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" event={"ID":"048fb8f1-e23d-40ae-bcd4-f873adf95ae8","Type":"ContainerStarted","Data":"63480dc52784ade3255f087fead2f7840ff654e64d01689d0f651a0c41c797af"} Nov 02 22:08:21.626196 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:21.626138 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podStartSLOduration=2.626121983 podStartE2EDuration="2.626121983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:21.573919626 +0000 UTC m=+2048.736014760" watchObservedRunningTime="2026-11-02 22:08:21.626121983 +0000 UTC m=+2048.788217102" Nov 02 22:08:22.167695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:22.167040 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:22.247694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:22.245963 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:22.408166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:22.408137 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:22.409028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:22.409009 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:08:23.510624 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:23.510591 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8"] Nov 02 22:08:23.514064 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:23.514035 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-204000-vhns8"] Nov 02 22:08:24.977173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:24.977137 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7303ebd5-c8b0-4831-828c-18c79c0ed25e" path="/var/lib/kubelet/pods/7303ebd5-c8b0-4831-828c-18c79c0ed25e/volumes" Nov 02 22:08:31.882442 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.882407 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4"] Nov 02 22:08:31.887897 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.887877 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:31.941114 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.941083 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:08:31.941114 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.941113 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:08:31.944842 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.944811 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4"] Nov 02 22:08:31.951871 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.951844 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:31.952018 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.951993 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:31.952113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.952100 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4zj\" (UniqueName: \"kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:31.989744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.989713 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:08:31.989878 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:31.989811 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:08:32.053008 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.052972 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.053171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.053069 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.053171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.053150 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4zj\" (UniqueName: \"kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.053756 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.053730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.055355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.055334 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.109631 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.109601 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4zj\" (UniqueName: \"kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.203771 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.203706 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:32.580310 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:32.580281 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4"] Nov 02 22:08:32.580974 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:08:32.580943 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb809bf_c829_4ecc_81a3_3367855b14e7.slice/crio-bcb79cd0197930d590130d1d16492f3f0de8116941d68b69d6b8134957cc6bbd WatchSource:0}: Error finding container bcb79cd0197930d590130d1d16492f3f0de8116941d68b69d6b8134957cc6bbd: Status 404 returned error can't find the container with id bcb79cd0197930d590130d1d16492f3f0de8116941d68b69d6b8134957cc6bbd Nov 02 22:08:33.490418 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:33.490391 5091 generic.go:356] "Generic (PLEG): container finished" podID="cdb809bf-c829-4ecc-81a3-3367855b14e7" containerID="e3f7fdfecc20e9312858813f7b995a92d62c26b6e65dada64023dd549db3f951" exitCode=0 Nov 02 22:08:33.490766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:33.490483 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" event={"ID":"cdb809bf-c829-4ecc-81a3-3367855b14e7","Type":"ContainerDied","Data":"e3f7fdfecc20e9312858813f7b995a92d62c26b6e65dada64023dd549db3f951"} Nov 02 22:08:33.490766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:33.490567 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" event={"ID":"cdb809bf-c829-4ecc-81a3-3367855b14e7","Type":"ContainerStarted","Data":"bcb79cd0197930d590130d1d16492f3f0de8116941d68b69d6b8134957cc6bbd"} Nov 02 22:08:34.657841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.657817 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:34.685226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.685204 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca\") pod \"cdb809bf-c829-4ecc-81a3-3367855b14e7\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " Nov 02 22:08:34.685349 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.685256 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj\" (UniqueName: \"kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj\") pod \"cdb809bf-c829-4ecc-81a3-3367855b14e7\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " Nov 02 22:08:34.685391 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.685366 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca\") pod \"cdb809bf-c829-4ecc-81a3-3367855b14e7\" (UID: \"cdb809bf-c829-4ecc-81a3-3367855b14e7\") " Nov 02 22:08:34.685555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.685531 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca" pod "cdb809bf-c829-4ecc-81a3-3367855b14e7" (UID: "cdb809bf-c829-4ecc-81a3-3367855b14e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:34.685970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.685951 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:34.687254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.687229 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca" pod "cdb809bf-c829-4ecc-81a3-3367855b14e7" (UID: "cdb809bf-c829-4ecc-81a3-3367855b14e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:34.687426 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.687404 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj" pod "cdb809bf-c829-4ecc-81a3-3367855b14e7" (UID: "cdb809bf-c829-4ecc-81a3-3367855b14e7"). InnerVolumeSpecName "kube-api-access-2h4zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:34.787210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.787140 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4zj\" (UniqueName: \"kubernetes.io/projected/cdb809bf-c829-4ecc-81a3-3367855b14e7-kube-api-access-2h4zj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:34.787210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:34.787170 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb809bf-c829-4ecc-81a3-3367855b14e7-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:35.505411 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:35.505379 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" Nov 02 22:08:35.505609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:35.505448 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4" event={"ID":"cdb809bf-c829-4ecc-81a3-3367855b14e7","Type":"ContainerDied","Data":"bcb79cd0197930d590130d1d16492f3f0de8116941d68b69d6b8134957cc6bbd"} Nov 02 22:08:35.505609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:35.505509 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb79cd0197930d590130d1d16492f3f0de8116941d68b69d6b8134957cc6bbd" Nov 02 22:08:37.112415 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:08:37.112381 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:08:50.214083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:50.214046 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4"] Nov 02 22:08:50.263400 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:50.263365 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3eb190-m4rs4"] Nov 02 22:08:50.977684 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:50.977648 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb809bf-c829-4ecc-81a3-3367855b14e7" path="/var/lib/kubelet/pods/cdb809bf-c829-4ecc-81a3-3367855b14e7/volumes" Nov 02 22:08:58.157127 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.157091 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5"] Nov 02 22:08:58.158633 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.158603 5091 cpu_manager.go:414] "containerMap: removing container" podUID="cdb809bf-c829-4ecc-81a3-3367855b14e7" containerName="revoke-keys" Nov 02 22:08:58.158829 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.158812 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb809bf-c829-4ecc-81a3-3367855b14e7" containerName="revoke-keys" Nov 02 22:08:58.159159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.159143 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb809bf-c829-4ecc-81a3-3367855b14e7" containerName="revoke-keys" Nov 02 22:08:58.180395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.180244 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.248750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.248709 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5"] Nov 02 22:08:58.277693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.277538 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdc145-openshift-default-dockercfg-945tf\"" Nov 02 22:08:58.358574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.358528 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.358930 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.358903 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.359066 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.359050 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.370718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.359196 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.371033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.371007 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.371266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.371249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.371407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.371393 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.371546 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.371526 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tph5\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.371658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.371644 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.473498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.473410 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.473815 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.473784 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.473974 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.473953 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.474136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.474116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.474311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.474292 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.474531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.474510 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.475401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.475374 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.475597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.475575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tph5\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.475791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.475772 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.475927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.474116 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.476023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.475441 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.476103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.475194 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.476872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.476779 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.477139 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.477113 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.479722 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.479700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.490017 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.489970 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.531319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.531288 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.532166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.532136 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tph5\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5\") pod \"e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.533911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.533888 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:08:58.997167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:58.997138 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5"] Nov 02 22:08:58.998198 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:08:58.998171 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfc1424_b3de_491f_a94f_c9408f14b660.slice/crio-c10c9abd05fa3cb30f16085bc24dfc51b1fa088ecdb31fe5842e91b9cdd5176f WatchSource:0}: Error finding container c10c9abd05fa3cb30f16085bc24dfc51b1fa088ecdb31fe5842e91b9cdd5176f: Status 404 returned error can't find the container with id c10c9abd05fa3cb30f16085bc24dfc51b1fa088ecdb31fe5842e91b9cdd5176f Nov 02 22:08:59.000772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:59.000740 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:08:59.000880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:59.000818 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:08:59.000880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:59.000861 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:08:59.693261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:59.693224 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" event={"ID":"1bfc1424-b3de-491f-a94f-c9408f14b660","Type":"ContainerStarted","Data":"2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61"} Nov 02 22:08:59.693709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:59.693282 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" event={"ID":"1bfc1424-b3de-491f-a94f-c9408f14b660","Type":"ContainerStarted","Data":"c10c9abd05fa3cb30f16085bc24dfc51b1fa088ecdb31fe5842e91b9cdd5176f"} Nov 02 22:08:59.860301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:08:59.860225 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podStartSLOduration=1.860207349 podStartE2EDuration="1.860207349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:59.858440748 +0000 UTC m=+2087.020535860" watchObservedRunningTime="2026-11-02 22:08:59.860207349 +0000 UTC m=+2087.022302471" Nov 02 22:09:00.534254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:00.534219 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:09:00.539835 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:00.539809 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:09:00.700144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:00.700111 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:09:00.701115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:00.701096 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:09:07.113093 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:09:07.113048 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:09:11.755077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.755039 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5"] Nov 02 22:09:11.760253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.760229 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870213 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870169 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870239 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870323 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzr2\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870373 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870479 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870569 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870758 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870638 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870758 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870753 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.870882 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.870802 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.921494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.921460 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e2be7cc3-openshift-default-dockercfg-82vtr\"" Nov 02 22:09:11.963649 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.963615 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5"] Nov 02 22:09:11.971697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.971655 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.971943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.971916 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972021 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972069 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972211 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972114 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzr2\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972211 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972171 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972326 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972326 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972368 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972397 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972826 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972804 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.972956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.972909 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.973316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.973296 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.973610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.973583 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.974408 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.974385 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:11.975597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:11.975573 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:12.029605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.029546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:12.029807 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.029788 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzr2\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:12.075243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.075218 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:12.503176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.503144 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5"] Nov 02 22:09:12.503456 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:09:12.503431 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e86e73_5891_4885_a3ed_e7ea709cddd6.slice/crio-64fcd6c86df4d7db4a29641d141edb1fc725ba24786efb3e519fe851c086fc71 WatchSource:0}: Error finding container 64fcd6c86df4d7db4a29641d141edb1fc725ba24786efb3e519fe851c086fc71: Status 404 returned error can't find the container with id 64fcd6c86df4d7db4a29641d141edb1fc725ba24786efb3e519fe851c086fc71 Nov 02 22:09:12.506232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.506203 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:12.506311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.506262 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:12.506311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.506288 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:12.797980 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.797888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" event={"ID":"c1e86e73-5891-4885-a3ed-e7ea709cddd6","Type":"ContainerStarted","Data":"d5b296797ce8c6b1b24191e21cc87e75a3451ae1e9e90c1e4225ef0294ef743c"} Nov 02 22:09:12.797980 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.797945 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" event={"ID":"c1e86e73-5891-4885-a3ed-e7ea709cddd6","Type":"ContainerStarted","Data":"64fcd6c86df4d7db4a29641d141edb1fc725ba24786efb3e519fe851c086fc71"} Nov 02 22:09:12.913758 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:12.913708 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podStartSLOduration=1.913662826 podStartE2EDuration="1.913662826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:12.911786942 +0000 UTC m=+2100.073882063" watchObservedRunningTime="2026-11-02 22:09:12.913662826 +0000 UTC m=+2100.075757945" Nov 02 22:09:13.084386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:13.077254 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:13.084386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:13.077374 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:09:13.084386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:13.077420 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 22:09:13.391784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:13.391753 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:09:13.397845 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:13.397823 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:09:14.088080 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:14.088039 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:09:14.088595 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:14.088138 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 22:09:15.078129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:15.078095 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:09:15.078435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:15.078394 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 22:09:16.083696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:16.083010 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:09:16.083696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:16.083082 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 22:09:17.085693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:17.084722 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:09:17.085693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:17.084786 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 22:09:18.087697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:18.083000 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:09:18.087697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:18.083062 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 22:09:19.120692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:19.120091 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:19.127692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:19.124085 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:19.145191 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:19.133933 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:09:20.161912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:20.161884 5091 scope.go:122] "RemoveContainer" containerID="9090b008a43731ea17b29e9ff6afa67588455ff4b922815b810715c1f30c1149" Nov 02 22:09:21.046812 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:21.046779 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8"] Nov 02 22:09:21.096169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:21.096133 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-224f53-lnxb8"] Nov 02 22:09:22.978135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:22.978092 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4d0f5f-17b5-46ec-b669-201516c5c6c2" path="/var/lib/kubelet/pods/0e4d0f5f-17b5-46ec-b669-201516c5c6c2/volumes" Nov 02 22:09:26.883779 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:26.883703 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd"] Nov 02 22:09:26.889777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:26.889757 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011227 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011509 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011302 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011509 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011413 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011509 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011471 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011526 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfggk\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011623 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011713 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.011962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.011828 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.097163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.097130 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd"] Nov 02 22:09:27.113774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.113740 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.113955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.113805 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.113955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.113862 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.113977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114052 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114204 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114097 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114204 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114164 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114211 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfggk\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114641 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114610 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.114957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.114933 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.115212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.115185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.115463 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.115440 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.117359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.117333 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.118250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.118226 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.170662 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.170590 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfggk\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.170815 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.170738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.204992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.204961 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:27.641446 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.641417 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd"] Nov 02 22:09:27.642260 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:09:27.642234 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf86efb_606d_4fc4_9b97_b7697e2216d2.slice/crio-a7e41b7ebce2fd0d01ed67cdfefcf4c169ec054e7ba9d7341df6e27f0a4e234d WatchSource:0}: Error finding container a7e41b7ebce2fd0d01ed67cdfefcf4c169ec054e7ba9d7341df6e27f0a4e234d: Status 404 returned error can't find the container with id a7e41b7ebce2fd0d01ed67cdfefcf4c169ec054e7ba9d7341df6e27f0a4e234d Nov 02 22:09:27.645637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.645591 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:27.645762 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.645721 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:27.645827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:27.645776 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:28.177704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:28.175131 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" event={"ID":"acf86efb-606d-4fc4-9b97-b7697e2216d2","Type":"ContainerStarted","Data":"44bef8d4d8e358829f200a8e86345e77eae4c6f5b813c0fa384414f2fa7dea17"} Nov 02 22:09:28.177704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:28.175199 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" event={"ID":"acf86efb-606d-4fc4-9b97-b7697e2216d2","Type":"ContainerStarted","Data":"a7e41b7ebce2fd0d01ed67cdfefcf4c169ec054e7ba9d7341df6e27f0a4e234d"} Nov 02 22:09:28.231702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:28.228751 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:28.253507 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:28.253473 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:28.253794 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:28.253760 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:28.405352 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:28.405290 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podStartSLOduration=2.405269313 podStartE2EDuration="2.405269313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:28.343121161 +0000 UTC m=+2115.505216286" watchObservedRunningTime="2026-11-02 22:09:28.405269313 +0000 UTC m=+2115.567364434" Nov 02 22:09:29.208695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:29.208394 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:29.208695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:29.208460 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:30.205913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:30.205876 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:30.206202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:30.206174 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:31.214695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:31.211132 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:31.214695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:31.211207 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:32.206693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:32.206048 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:32.206693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:32.206123 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:33.206693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:33.206034 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:33.206693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:33.206100 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:34.208695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:34.205970 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 22:09:34.208695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:34.206065 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 22:09:35.299841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:35.299810 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:35.301072 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:35.301042 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:35.301924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:35.301898 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:09:37.121282 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:09:37.121244 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:09:38.316359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:38.316330 5091 generic.go:356] "Generic (PLEG): container finished" podID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerID="20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f" exitCode=0 Nov 02 22:09:38.316766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:38.316405 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" event={"ID":"2ae31de7-5263-4371-b452-ebb1b1edc4de","Type":"ContainerDied","Data":"20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f"} Nov 02 22:09:38.316838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:38.316823 5091 scope.go:122] "RemoveContainer" containerID="20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f" Nov 02 22:09:38.318665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:38.318639 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:38.318770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:38.318726 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:38.318810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:38.318784 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:39.326188 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:39.326156 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" event={"ID":"2ae31de7-5263-4371-b452-ebb1b1edc4de","Type":"ContainerStarted","Data":"38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c"} Nov 02 22:09:39.652912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:39.652884 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:09:39.653127 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:39.653057 5091 patch_prober.go:29] interesting pod/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:09:39.653127 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:39.653104 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 22:09:40.666702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:40.662007 5091 patch_prober.go:29] interesting pod/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:09:40.666702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:40.662071 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 22:09:41.652623 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:41.652585 5091 patch_prober.go:29] interesting pod/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:09:41.652827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:41.652635 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 22:09:41.991235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:41.991142 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g"] Nov 02 22:09:41.995630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:41.995613 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.046717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.046686 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g"] Nov 02 22:09:42.049832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.049808 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.049956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.049866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.049956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.049885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.050065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.049982 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgfm\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.050123 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.050092 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.050177 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.050143 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.050239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.050225 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.050291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.050271 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.050291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.050286 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151355 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgfm\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151400 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151489 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151652 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151747 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.151908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151779 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.152095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151946 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.152095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.151977 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.152095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.152043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.152202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.152141 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.152449 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.152420 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.154572 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.154545 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.154780 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.154757 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.207872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.207839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.208018 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.207961 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgfm\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.310241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.309974 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:42.653515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.653486 5091 patch_prober.go:29] interesting pod/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:09:42.653716 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.653536 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 22:09:42.801169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.801116 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g"] Nov 02 22:09:42.817316 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:09:42.816787 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c36ca8_9b66_4235_8531_4e7e1343a475.slice/crio-54914e2ab2abaf71ac0c89c0b39ff19145574bc6485d1332b3a78f5e82d7548a WatchSource:0}: Error finding container 54914e2ab2abaf71ac0c89c0b39ff19145574bc6485d1332b3a78f5e82d7548a: Status 404 returned error can't find the container with id 54914e2ab2abaf71ac0c89c0b39ff19145574bc6485d1332b3a78f5e82d7548a Nov 02 22:09:42.819811 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.819777 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:42.819915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.819846 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:42.819915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:42.819877 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:43.411477 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:43.411439 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" event={"ID":"91c36ca8-9b66-4235-8531-4e7e1343a475","Type":"ContainerStarted","Data":"429780485c77c638766ce9f824d89fdd6406adfc9f8e5f8815ad1df357487ebe"} Nov 02 22:09:43.411950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:43.411504 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" event={"ID":"91c36ca8-9b66-4235-8531-4e7e1343a475","Type":"ContainerStarted","Data":"54914e2ab2abaf71ac0c89c0b39ff19145574bc6485d1332b3a78f5e82d7548a"} Nov 02 22:09:43.553213 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:43.553161 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podStartSLOduration=2.553147385 podStartE2EDuration="2.553147385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:43.551086266 +0000 UTC m=+2130.713181399" watchObservedRunningTime="2026-11-02 22:09:43.553147385 +0000 UTC m=+2130.715242505" Nov 02 22:09:43.653121 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:43.653089 5091 patch_prober.go:29] interesting pod/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:09:43.653341 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:43.653147 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 02 22:09:44.311782 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:44.311075 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:44.313656 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:44.313619 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 22:09:44.313925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:44.313897 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 22:09:44.698321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:44.698290 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:09:44.699275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:44.699247 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:09:44.700215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:44.700194 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:09:45.310734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:45.310696 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 22:09:45.310912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:45.310765 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 22:09:46.310516 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:46.310475 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 22:09:46.311070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:46.310538 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 22:09:47.310744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:47.310709 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 22:09:47.311166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:47.310762 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 22:09:48.310962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:48.310926 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 22:09:48.311351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:48.310986 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 22:09:49.310561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:49.310522 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 22:09:49.310775 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:49.310595 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 22:09:50.315556 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:50.315523 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:50.316071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:50.315913 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:09:50.316457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:09:50.316436 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:10:00.046558 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:00.046520 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f"] Nov 02 22:10:00.064936 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:00.064911 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484614ae-r6f8f"] Nov 02 22:10:00.977468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:00.977424 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1357ea-b0e5-407a-8d86-3082636a24f0" path="/var/lib/kubelet/pods/1f1357ea-b0e5-407a-8d86-3082636a24f0/volumes" Nov 02 22:10:03.536428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.536392 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27"] Nov 02 22:10:03.542238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.542210 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.597757 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.597728 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27"] Nov 02 22:10:03.623866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.623830 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wtp\" (UniqueName: \"kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.624044 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.623881 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.624138 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.624116 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.641948 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.641918 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:03.642064 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.641986 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:10:03.654173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.654136 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:03.691838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.691816 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:03.725062 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.725038 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wtp\" (UniqueName: \"kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.725167 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.725081 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.725209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.725189 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.726379 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.726351 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.737189 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.737161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.781021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.780993 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wtp\" (UniqueName: \"kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp\") pod \"maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:03.857568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:03.857493 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:04.235921 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.235888 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27"] Nov 02 22:10:04.237097 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:10:04.237067 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d780167_d3f7_41a2_8a64_fbba78fed01f.slice/crio-fab16462e857a6d7ee6010ef4d4940e5a14acea4eeb22a713eabbeae4bb42eeb WatchSource:0}: Error finding container fab16462e857a6d7ee6010ef4d4940e5a14acea4eeb22a713eabbeae4bb42eeb: Status 404 returned error can't find the container with id fab16462e857a6d7ee6010ef4d4940e5a14acea4eeb22a713eabbeae4bb42eeb Nov 02 22:10:04.574030 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.573858 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" event={"ID":"0d780167-d3f7-41a2-8a64-fbba78fed01f","Type":"ContainerStarted","Data":"fab16462e857a6d7ee6010ef4d4940e5a14acea4eeb22a713eabbeae4bb42eeb"} Nov 02 22:10:04.721749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.721715 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8"] Nov 02 22:10:04.726085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.726068 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.781304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.781268 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8"] Nov 02 22:10:04.845764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.845651 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.845764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.845707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkvd\" (UniqueName: \"kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.845955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.845816 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.946969 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.946928 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.946969 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.946970 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkvd\" (UniqueName: \"kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.947248 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.947013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.947708 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.947687 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:04.949479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:04.949443 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:05.003875 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:05.003839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkvd\" (UniqueName: \"kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd\") pod \"maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:05.052832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:05.052802 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:05.522948 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:05.522916 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8"] Nov 02 22:10:05.523155 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:10:05.523126 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b804a1a_0784_455e_93c4_ab084619e865.slice/crio-bfc764511de25dfc6c9065698902ce89720cad482a5c7127bcbec1e540d80360 WatchSource:0}: Error finding container bfc764511de25dfc6c9065698902ce89720cad482a5c7127bcbec1e540d80360: Status 404 returned error can't find the container with id bfc764511de25dfc6c9065698902ce89720cad482a5c7127bcbec1e540d80360 Nov 02 22:10:05.584340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:05.584310 5091 generic.go:356] "Generic (PLEG): container finished" podID="0d780167-d3f7-41a2-8a64-fbba78fed01f" containerID="1e00140b948d84c6599b0ec986fd73ba17f0d688152b86ba29ac02120cc1e501" exitCode=0 Nov 02 22:10:05.584766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:05.584398 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" event={"ID":"0d780167-d3f7-41a2-8a64-fbba78fed01f","Type":"ContainerDied","Data":"1e00140b948d84c6599b0ec986fd73ba17f0d688152b86ba29ac02120cc1e501"} Nov 02 22:10:05.585976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:05.585954 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" event={"ID":"8b804a1a-0784-455e-93c4-ab084619e865","Type":"ContainerStarted","Data":"bfc764511de25dfc6c9065698902ce89720cad482a5c7127bcbec1e540d80360"} Nov 02 22:10:06.594280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.594246 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b804a1a-0784-455e-93c4-ab084619e865" containerID="a3f096f50904f4e4df477b839cd604d48410bb41a9d40c36b2b5d56dda1b77bf" exitCode=0 Nov 02 22:10:06.594707 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.594331 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" event={"ID":"8b804a1a-0784-455e-93c4-ab084619e865","Type":"ContainerDied","Data":"a3f096f50904f4e4df477b839cd604d48410bb41a9d40c36b2b5d56dda1b77bf"} Nov 02 22:10:06.755466 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.755440 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:06.878682 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.878643 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca\") pod \"0d780167-d3f7-41a2-8a64-fbba78fed01f\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " Nov 02 22:10:06.878931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.878736 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp\" (UniqueName: \"kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp\") pod \"0d780167-d3f7-41a2-8a64-fbba78fed01f\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " Nov 02 22:10:06.879734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.879706 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca\") pod \"0d780167-d3f7-41a2-8a64-fbba78fed01f\" (UID: \"0d780167-d3f7-41a2-8a64-fbba78fed01f\") " Nov 02 22:10:06.880041 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.880009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca" pod "0d780167-d3f7-41a2-8a64-fbba78fed01f" (UID: "0d780167-d3f7-41a2-8a64-fbba78fed01f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:06.881103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.880612 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:06.882876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.882841 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp" pod "0d780167-d3f7-41a2-8a64-fbba78fed01f" (UID: "0d780167-d3f7-41a2-8a64-fbba78fed01f"). InnerVolumeSpecName "kube-api-access-m8wtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:06.884299 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.884268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca" pod "0d780167-d3f7-41a2-8a64-fbba78fed01f" (UID: "0d780167-d3f7-41a2-8a64-fbba78fed01f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:06.981944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.981918 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d780167-d3f7-41a2-8a64-fbba78fed01f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:06.981944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:06.981942 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wtp\" (UniqueName: \"kubernetes.io/projected/0d780167-d3f7-41a2-8a64-fbba78fed01f-kube-api-access-m8wtp\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:07.115638 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:10:07.115596 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:10:07.603897 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.603863 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" Nov 02 22:10:07.604424 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.603898 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27" event={"ID":"0d780167-d3f7-41a2-8a64-fbba78fed01f","Type":"ContainerDied","Data":"fab16462e857a6d7ee6010ef4d4940e5a14acea4eeb22a713eabbeae4bb42eeb"} Nov 02 22:10:07.604424 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.603989 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fab16462e857a6d7ee6010ef4d4940e5a14acea4eeb22a713eabbeae4bb42eeb" Nov 02 22:10:07.773988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.773965 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:07.892923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.892889 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca\") pod \"8b804a1a-0784-455e-93c4-ab084619e865\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " Nov 02 22:10:07.893105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.893083 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca\") pod \"8b804a1a-0784-455e-93c4-ab084619e865\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " Nov 02 22:10:07.893178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.893162 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd\" (UniqueName: \"kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd\") pod \"8b804a1a-0784-455e-93c4-ab084619e865\" (UID: \"8b804a1a-0784-455e-93c4-ab084619e865\") " Nov 02 22:10:07.893246 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.893204 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca" pod "8b804a1a-0784-455e-93c4-ab084619e865" (UID: "8b804a1a-0784-455e-93c4-ab084619e865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:07.893614 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.893595 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:07.895117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.895096 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca" pod "8b804a1a-0784-455e-93c4-ab084619e865" (UID: "8b804a1a-0784-455e-93c4-ab084619e865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:07.895311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.895289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd" pod "8b804a1a-0784-455e-93c4-ab084619e865" (UID: "8b804a1a-0784-455e-93c4-ab084619e865"). InnerVolumeSpecName "kube-api-access-4zkvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:07.995425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.995391 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b804a1a-0784-455e-93c4-ab084619e865-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:07.995425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:07.995420 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkvd\" (UniqueName: \"kubernetes.io/projected/8b804a1a-0784-455e-93c4-ab084619e865-kube-api-access-4zkvd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:08.612283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:08.612255 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" Nov 02 22:10:08.612767 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:08.612285 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8" event={"ID":"8b804a1a-0784-455e-93c4-ab084619e865","Type":"ContainerDied","Data":"bfc764511de25dfc6c9065698902ce89720cad482a5c7127bcbec1e540d80360"} Nov 02 22:10:08.612767 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:08.612337 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfc764511de25dfc6c9065698902ce89720cad482a5c7127bcbec1e540d80360" Nov 02 22:10:18.049035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:18.048995 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk"] Nov 02 22:10:18.098190 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:18.098154 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee35b83a-ttmnk"] Nov 02 22:10:18.980885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:18.980847 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe620b5-b6cf-444b-8fd7-f3e5d021e317" path="/var/lib/kubelet/pods/9fe620b5-b6cf-444b-8fd7-f3e5d021e317/volumes" Nov 02 22:10:20.346650 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.346617 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw"] Nov 02 22:10:20.347277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.347261 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b804a1a-0784-455e-93c4-ab084619e865" containerName="revoke-keys" Nov 02 22:10:20.347339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.347279 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b804a1a-0784-455e-93c4-ab084619e865" containerName="revoke-keys" Nov 02 22:10:20.347339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.347300 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0d780167-d3f7-41a2-8a64-fbba78fed01f" containerName="revoke-keys" Nov 02 22:10:20.347339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.347305 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d780167-d3f7-41a2-8a64-fbba78fed01f" containerName="revoke-keys" Nov 02 22:10:20.347437 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.347403 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b804a1a-0784-455e-93c4-ab084619e865" containerName="revoke-keys" Nov 02 22:10:20.347437 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.347415 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d780167-d3f7-41a2-8a64-fbba78fed01f" containerName="revoke-keys" Nov 02 22:10:20.352298 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.352282 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.407349 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.407324 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw"] Nov 02 22:10:20.451958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.451928 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:20.452087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.452009 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:10:20.452087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.452020 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:20.453534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.453518 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:20.525023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.524980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.525188 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.525170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8698v\" (UniqueName: \"kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.525229 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.525209 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.626396 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.626356 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8698v\" (UniqueName: \"kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.626561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.626416 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.626561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.626499 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.627094 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.627067 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.628941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.628916 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.682412 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.682384 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8698v\" (UniqueName: \"kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v\") pod \"maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:20.965927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:20.965849 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:21.657154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:21.657108 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw"] Nov 02 22:10:21.675797 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:21.675761 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8"] Nov 02 22:10:21.679349 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:21.679314 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-588b5c-d6rz8"] Nov 02 22:10:21.781304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:21.781198 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" event={"ID":"4f5026bf-03b1-4062-a820-807c847d79a2","Type":"ContainerStarted","Data":"ae42c6d68de53ef91ccea5cbfa990929c1ef5a0afe99b226e4d1c978bb47522c"} Nov 02 22:10:22.789999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:22.789963 5091 generic.go:356] "Generic (PLEG): container finished" podID="4f5026bf-03b1-4062-a820-807c847d79a2" containerID="c24b9f3afc91adc420d3f0cb31fda5b84b01afc530cf6dbcdc370f26a6bf6b4b" exitCode=0 Nov 02 22:10:22.790365 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:22.790039 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" event={"ID":"4f5026bf-03b1-4062-a820-807c847d79a2","Type":"ContainerDied","Data":"c24b9f3afc91adc420d3f0cb31fda5b84b01afc530cf6dbcdc370f26a6bf6b4b"} Nov 02 22:10:22.978519 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:22.978440 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b804a1a-0784-455e-93c4-ab084619e865" path="/var/lib/kubelet/pods/8b804a1a-0784-455e-93c4-ab084619e865/volumes" Nov 02 22:10:23.962076 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:23.962051 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:24.029707 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.029662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v\" (UniqueName: \"kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v\") pod \"4f5026bf-03b1-4062-a820-807c847d79a2\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " Nov 02 22:10:24.029879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.029723 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca\") pod \"4f5026bf-03b1-4062-a820-807c847d79a2\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " Nov 02 22:10:24.029879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.029828 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca\") pod \"4f5026bf-03b1-4062-a820-807c847d79a2\" (UID: \"4f5026bf-03b1-4062-a820-807c847d79a2\") " Nov 02 22:10:24.030205 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.030182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca" pod "4f5026bf-03b1-4062-a820-807c847d79a2" (UID: "4f5026bf-03b1-4062-a820-807c847d79a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:24.030428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.030404 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:24.032084 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.032054 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v" pod "4f5026bf-03b1-4062-a820-807c847d79a2" (UID: "4f5026bf-03b1-4062-a820-807c847d79a2"). InnerVolumeSpecName "kube-api-access-8698v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:24.032370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.032348 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca" pod "4f5026bf-03b1-4062-a820-807c847d79a2" (UID: "4f5026bf-03b1-4062-a820-807c847d79a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:24.131433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.131399 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8698v\" (UniqueName: \"kubernetes.io/projected/4f5026bf-03b1-4062-a820-807c847d79a2-kube-api-access-8698v\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:24.131433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.131433 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5026bf-03b1-4062-a820-807c847d79a2-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:24.806910 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.806878 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" Nov 02 22:10:24.807090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.806913 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw" event={"ID":"4f5026bf-03b1-4062-a820-807c847d79a2","Type":"ContainerDied","Data":"ae42c6d68de53ef91ccea5cbfa990929c1ef5a0afe99b226e4d1c978bb47522c"} Nov 02 22:10:24.807090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:24.806966 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae42c6d68de53ef91ccea5cbfa990929c1ef5a0afe99b226e4d1c978bb47522c" Nov 02 22:10:34.965254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:34.965174 5091 generic.go:356] "Generic (PLEG): container finished" podID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerID="b33bb3a1baf2de332c1888ce7c3659f5e968d9a4d615c07a2447b0dcb192f915" exitCode=0 Nov 02 22:10:34.965688 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:34.965255 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" event={"ID":"048fb8f1-e23d-40ae-bcd4-f873adf95ae8","Type":"ContainerDied","Data":"b33bb3a1baf2de332c1888ce7c3659f5e968d9a4d615c07a2447b0dcb192f915"} Nov 02 22:10:34.965750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:34.965708 5091 scope.go:122] "RemoveContainer" containerID="b33bb3a1baf2de332c1888ce7c3659f5e968d9a4d615c07a2447b0dcb192f915" Nov 02 22:10:34.967347 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:34.967320 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:34.967416 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:34.967390 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:34.967453 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:34.967423 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:35.992710 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:35.992645 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" event={"ID":"048fb8f1-e23d-40ae-bcd4-f873adf95ae8","Type":"ContainerStarted","Data":"43e19846384ce5cda96733c6ff5dbde027f6162a9380aa18610c350ab1ea74c5"} Nov 02 22:10:36.189727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:36.185057 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:10:36.194843 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:36.194812 5091 patch_prober.go:29] interesting pod/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:10:36.195015 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:36.194889 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 22:10:37.116197 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:10:37.116162 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:10:37.191694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:37.190816 5091 patch_prober.go:29] interesting pod/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:10:37.191694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:37.190876 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 22:10:37.408584 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:37.408547 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:10:38.171694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:38.165392 5091 patch_prober.go:29] interesting pod/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:10:38.171694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:38.165451 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 22:10:39.186703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:39.186569 5091 patch_prober.go:29] interesting pod/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:10:39.186703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:39.186632 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 22:10:40.169089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:40.169050 5091 patch_prober.go:29] interesting pod/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:10:40.170071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:40.170037 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 22:10:41.171697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:41.168059 5091 patch_prober.go:29] interesting pod/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:10:41.171697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:41.168122 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 22:10:42.170380 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:42.170349 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:10:42.171389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:42.171367 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:10:58.330987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:58.330905 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh"] Nov 02 22:10:58.379916 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:58.379886 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc235-rnrzh"] Nov 02 22:10:58.980146 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:10:58.980108 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192ff04e-2819-46fe-bc68-84c0113873df" path="/var/lib/kubelet/pods/192ff04e-2819-46fe-bc68-84c0113873df/volumes" Nov 02 22:11:04.539436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.539401 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs"] Nov 02 22:11:04.540102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.540087 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4f5026bf-03b1-4062-a820-807c847d79a2" containerName="revoke-keys" Nov 02 22:11:04.540150 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.540104 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5026bf-03b1-4062-a820-807c847d79a2" containerName="revoke-keys" Nov 02 22:11:04.540206 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.540197 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5026bf-03b1-4062-a820-807c847d79a2" containerName="revoke-keys" Nov 02 22:11:04.542877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.542862 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.610375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.610340 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.610589 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.610511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgw8\" (UniqueName: \"kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.610589 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.610559 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.644547 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.644511 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:11:04.644746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.644553 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:11:04.644746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.644608 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:11:04.644746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.644613 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:11:04.648876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.648851 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs"] Nov 02 22:11:04.712147 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.712116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.712268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.712198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgw8\" (UniqueName: \"kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.712268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.712221 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.712826 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.712807 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.714605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.714587 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.768100 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.768076 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgw8\" (UniqueName: \"kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8\") pod \"maas-api-revoke-keys-e2e-api-b-587629-vmshs\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:04.855883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:04.855823 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:05.231835 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:05.231809 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs"] Nov 02 22:11:05.232574 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:11:05.232550 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff92398_fa9c_4df1_ab68_a33649d3d2f8.slice/crio-9a4247f0a398b88d1be22afd52dfef9031086172f00200c4521a2416d0c6c3d8 WatchSource:0}: Error finding container 9a4247f0a398b88d1be22afd52dfef9031086172f00200c4521a2416d0c6c3d8: Status 404 returned error can't find the container with id 9a4247f0a398b88d1be22afd52dfef9031086172f00200c4521a2416d0c6c3d8 Nov 02 22:11:05.347742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:05.347710 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" event={"ID":"1ff92398-fa9c-4df1-ab68-a33649d3d2f8","Type":"ContainerStarted","Data":"9a4247f0a398b88d1be22afd52dfef9031086172f00200c4521a2416d0c6c3d8"} Nov 02 22:11:06.356823 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:06.356790 5091 generic.go:356] "Generic (PLEG): container finished" podID="1ff92398-fa9c-4df1-ab68-a33649d3d2f8" containerID="3eecc6be635b1c8e3cd5146c565629e952b66c2cded519e641eec0d1b52ef7bc" exitCode=0 Nov 02 22:11:06.357240 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:06.356870 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" event={"ID":"1ff92398-fa9c-4df1-ab68-a33649d3d2f8","Type":"ContainerDied","Data":"3eecc6be635b1c8e3cd5146c565629e952b66c2cded519e641eec0d1b52ef7bc"} Nov 02 22:11:07.112854 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:11:07.112815 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:11:07.630140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.630115 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:07.769773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.762230 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8\" (UniqueName: \"kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8\") pod \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " Nov 02 22:11:07.769773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.762285 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca\") pod \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " Nov 02 22:11:07.769773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.762491 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca\") pod \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\" (UID: \"1ff92398-fa9c-4df1-ab68-a33649d3d2f8\") " Nov 02 22:11:07.769773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.763802 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca" pod "1ff92398-fa9c-4df1-ab68-a33649d3d2f8" (UID: "1ff92398-fa9c-4df1-ab68-a33649d3d2f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:07.769773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.765883 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca" pod "1ff92398-fa9c-4df1-ab68-a33649d3d2f8" (UID: "1ff92398-fa9c-4df1-ab68-a33649d3d2f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:07.769773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.766166 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8" pod "1ff92398-fa9c-4df1-ab68-a33649d3d2f8" (UID: "1ff92398-fa9c-4df1-ab68-a33649d3d2f8"). InnerVolumeSpecName "kube-api-access-wmgw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:07.865343 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.865293 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmgw8\" (UniqueName: \"kubernetes.io/projected/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-kube-api-access-wmgw8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:07.865343 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.865345 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:07.865567 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:07.865362 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff92398-fa9c-4df1-ab68-a33649d3d2f8-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:08.465693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:08.464658 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" event={"ID":"1ff92398-fa9c-4df1-ab68-a33649d3d2f8","Type":"ContainerDied","Data":"9a4247f0a398b88d1be22afd52dfef9031086172f00200c4521a2416d0c6c3d8"} Nov 02 22:11:08.465693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:08.464771 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a4247f0a398b88d1be22afd52dfef9031086172f00200c4521a2416d0c6c3d8" Nov 02 22:11:08.465693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:08.464873 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs" Nov 02 22:11:20.203366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:20.203331 5091 scope.go:122] "RemoveContainer" containerID="a7c9a7b798db2c58817d7fad4317e47691a7b4c29d2d29e7bd806f0c3bdd3621" Nov 02 22:11:20.218418 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:20.218395 5091 scope.go:122] "RemoveContainer" containerID="c26e3a4c4b0d80330f19ecf258b33140c2f6015aec549e72139a79644ea80007" Nov 02 22:11:20.227806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:20.227776 5091 scope.go:122] "RemoveContainer" containerID="ed70ce54beb2dcbf2d184f17bcbb227afbe0e0552648ee2d39537dbd4bbdfc4d" Nov 02 22:11:28.351061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.351029 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp"] Nov 02 22:11:28.351639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.351626 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1ff92398-fa9c-4df1-ab68-a33649d3d2f8" containerName="revoke-keys" Nov 02 22:11:28.351701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.351641 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff92398-fa9c-4df1-ab68-a33649d3d2f8" containerName="revoke-keys" Nov 02 22:11:28.351793 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.351783 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff92398-fa9c-4df1-ab68-a33649d3d2f8" containerName="revoke-keys" Nov 02 22:11:28.356911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.356893 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.410634 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.410605 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp"] Nov 02 22:11:28.455590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.455562 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:11:28.455846 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.455782 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:11:28.456004 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.455864 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:11:28.456004 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.455879 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:11:28.524144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.524114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98x9f\" (UniqueName: \"kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.524395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.524372 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.524489 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.524417 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.625746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.625717 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98x9f\" (UniqueName: \"kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.625994 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.625968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.626082 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.626019 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.626578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.626555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.628351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.628329 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.682074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.682050 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98x9f\" (UniqueName: \"kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f\") pod \"maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:28.970529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:28.970454 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:29.349387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:29.349356 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp"] Nov 02 22:11:29.350317 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:11:29.350290 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad4a9a1_b65e_4cdb_b797_da7ece8ba4bb.slice/crio-15bb0d1f8f5830bae889584f5a726ddd8001277c890eddea684182cfe458fcca WatchSource:0}: Error finding container 15bb0d1f8f5830bae889584f5a726ddd8001277c890eddea684182cfe458fcca: Status 404 returned error can't find the container with id 15bb0d1f8f5830bae889584f5a726ddd8001277c890eddea684182cfe458fcca Nov 02 22:11:29.705131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:29.705101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" event={"ID":"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb","Type":"ContainerStarted","Data":"909704827f646e8cb369227343d24156f0b82be96cd1a23cd68fa3b3af599551"} Nov 02 22:11:29.705498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:29.705163 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" event={"ID":"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb","Type":"ContainerStarted","Data":"15bb0d1f8f5830bae889584f5a726ddd8001277c890eddea684182cfe458fcca"} Nov 02 22:11:29.817883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:29.817834 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" podStartSLOduration=1.817819466 podStartE2EDuration="1.817819466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:29.815731257 +0000 UTC m=+2236.977826377" watchObservedRunningTime="2026-11-02 22:11:29.817819466 +0000 UTC m=+2236.979914586" Nov 02 22:11:30.713822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:30.713785 5091 generic.go:356] "Generic (PLEG): container finished" podID="bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" containerID="909704827f646e8cb369227343d24156f0b82be96cd1a23cd68fa3b3af599551" exitCode=0 Nov 02 22:11:30.714223 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:30.713863 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" event={"ID":"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb","Type":"ContainerDied","Data":"909704827f646e8cb369227343d24156f0b82be96cd1a23cd68fa3b3af599551"} Nov 02 22:11:31.895110 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:31.895085 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:32.070211 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.070137 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f\" (UniqueName: \"kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f\") pod \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " Nov 02 22:11:32.070358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.070304 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca\") pod \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " Nov 02 22:11:32.070417 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.070394 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca\") pod \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\" (UID: \"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb\") " Nov 02 22:11:32.070800 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.070774 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca" pod "bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" (UID: "bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:32.071040 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.071014 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:32.072191 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.072174 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f" pod "bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" (UID: "bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb"). InnerVolumeSpecName "kube-api-access-98x9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:32.072575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.072556 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca" pod "bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" (UID: "bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:32.172630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.172601 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98x9f\" (UniqueName: \"kubernetes.io/projected/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-kube-api-access-98x9f\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:32.172630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.172632 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:32.729770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.729739 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" Nov 02 22:11:32.729945 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.729780 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp" event={"ID":"bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb","Type":"ContainerDied","Data":"15bb0d1f8f5830bae889584f5a726ddd8001277c890eddea684182cfe458fcca"} Nov 02 22:11:32.729945 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:32.729821 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15bb0d1f8f5830bae889584f5a726ddd8001277c890eddea684182cfe458fcca" Nov 02 22:11:37.114844 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:11:37.114796 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:11:42.048246 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:42.048212 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8"] Nov 02 22:11:42.100759 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:42.100721 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122c16b-vfhl8"] Nov 02 22:11:42.978925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:42.978892 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34125d68-c999-4549-90b0-80630fcf8407" path="/var/lib/kubelet/pods/34125d68-c999-4549-90b0-80630fcf8407/volumes" Nov 02 22:11:43.487627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:43.487591 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27"] Nov 02 22:11:43.535300 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:43.535268 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2be7cc3-5dz27"] Nov 02 22:11:44.977314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:11:44.977271 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d780167-d3f7-41a2-8a64-fbba78fed01f" path="/var/lib/kubelet/pods/0d780167-d3f7-41a2-8a64-fbba78fed01f/volumes" Nov 02 22:12:04.259124 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:04.259093 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw"] Nov 02 22:12:04.308324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:04.308291 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2050f3-s2ljw"] Nov 02 22:12:04.980494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:04.980456 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5026bf-03b1-4062-a820-807c847d79a2" path="/var/lib/kubelet/pods/4f5026bf-03b1-4062-a820-807c847d79a2/volumes" Nov 02 22:12:06.637729 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.637693 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75"] Nov 02 22:12:06.638360 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.638344 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" containerName="revoke-keys" Nov 02 22:12:06.638408 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.638361 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" containerName="revoke-keys" Nov 02 22:12:06.638516 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.638505 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" containerName="revoke-keys" Nov 02 22:12:06.643467 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.643438 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.688635 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.688606 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.688878 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.688858 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98qx\" (UniqueName: \"kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.688941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.688904 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.695744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.695718 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:06.698125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.698103 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75"] Nov 02 22:12:06.742942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.742912 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:06.743125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.742950 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:06.745341 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.745315 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:12:06.790227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.790198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98qx\" (UniqueName: \"kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.790373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.790255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.790373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.790301 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.790941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.790917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.792691 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.792658 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.846583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.846554 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98qx\" (UniqueName: \"kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx\") pod \"maas-api-revoke-keys-e2e-api-a-bdc145-mvd75\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:06.958346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:06.958266 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:07.115177 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:12:07.115135 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:12:07.335699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:07.335651 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75"] Nov 02 22:12:07.336928 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:12:07.336892 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b12d95_3e27_40d0_a989_dd11fb78de5f.slice/crio-612478796e923ec9b7db2bb78f316b1451d80d6a2add986f8d5dbd906df0ccc3 WatchSource:0}: Error finding container 612478796e923ec9b7db2bb78f316b1451d80d6a2add986f8d5dbd906df0ccc3: Status 404 returned error can't find the container with id 612478796e923ec9b7db2bb78f316b1451d80d6a2add986f8d5dbd906df0ccc3 Nov 02 22:12:08.055432 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:08.055384 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" event={"ID":"92b12d95-3e27-40d0-a989-dd11fb78de5f","Type":"ContainerStarted","Data":"d2b2533f051637fa3b33bbd14099a15507a90e7a003adeeab41c06082ad4e914"} Nov 02 22:12:08.055929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:08.055447 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" event={"ID":"92b12d95-3e27-40d0-a989-dd11fb78de5f","Type":"ContainerStarted","Data":"612478796e923ec9b7db2bb78f316b1451d80d6a2add986f8d5dbd906df0ccc3"} Nov 02 22:12:08.168308 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:08.168259 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" podStartSLOduration=2.168244024 podStartE2EDuration="2.168244024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:08.166629684 +0000 UTC m=+2275.328724804" watchObservedRunningTime="2026-11-02 22:12:08.168244024 +0000 UTC m=+2275.330339172" Nov 02 22:12:09.076608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:09.075846 5091 generic.go:356] "Generic (PLEG): container finished" podID="92b12d95-3e27-40d0-a989-dd11fb78de5f" containerID="d2b2533f051637fa3b33bbd14099a15507a90e7a003adeeab41c06082ad4e914" exitCode=0 Nov 02 22:12:09.076608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:09.075962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" event={"ID":"92b12d95-3e27-40d0-a989-dd11fb78de5f","Type":"ContainerDied","Data":"d2b2533f051637fa3b33bbd14099a15507a90e7a003adeeab41c06082ad4e914"} Nov 02 22:12:10.244683 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.244647 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:10.347344 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.347265 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca\") pod \"92b12d95-3e27-40d0-a989-dd11fb78de5f\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " Nov 02 22:12:10.347501 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.347428 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca\") pod \"92b12d95-3e27-40d0-a989-dd11fb78de5f\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " Nov 02 22:12:10.347501 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.347457 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx\" (UniqueName: \"kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx\") pod \"92b12d95-3e27-40d0-a989-dd11fb78de5f\" (UID: \"92b12d95-3e27-40d0-a989-dd11fb78de5f\") " Nov 02 22:12:10.347649 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.347624 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca" pod "92b12d95-3e27-40d0-a989-dd11fb78de5f" (UID: "92b12d95-3e27-40d0-a989-dd11fb78de5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:10.347943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.347923 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.349591 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.349570 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca" pod "92b12d95-3e27-40d0-a989-dd11fb78de5f" (UID: "92b12d95-3e27-40d0-a989-dd11fb78de5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:10.349663 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.349634 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx" pod "92b12d95-3e27-40d0-a989-dd11fb78de5f" (UID: "92b12d95-3e27-40d0-a989-dd11fb78de5f"). InnerVolumeSpecName "kube-api-access-n98qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:10.449375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.449343 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b12d95-3e27-40d0-a989-dd11fb78de5f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.449375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.449371 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98qx\" (UniqueName: \"kubernetes.io/projected/92b12d95-3e27-40d0-a989-dd11fb78de5f-kube-api-access-n98qx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.709834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.709792 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv"] Nov 02 22:12:10.710066 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:10.710038 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" containerID="cri-o://38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c" gracePeriod=30 Nov 02 22:12:11.093514 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:11.093482 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" Nov 02 22:12:11.093514 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:11.093502 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75" event={"ID":"92b12d95-3e27-40d0-a989-dd11fb78de5f","Type":"ContainerDied","Data":"612478796e923ec9b7db2bb78f316b1451d80d6a2add986f8d5dbd906df0ccc3"} Nov 02 22:12:11.093784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:11.093558 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="612478796e923ec9b7db2bb78f316b1451d80d6a2add986f8d5dbd906df0ccc3" Nov 02 22:12:12.805108 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.805074 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg"] Nov 02 22:12:12.805752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.805734 5091 cpu_manager.go:414] "containerMap: removing container" podUID="92b12d95-3e27-40d0-a989-dd11fb78de5f" containerName="revoke-keys" Nov 02 22:12:12.805806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.805754 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b12d95-3e27-40d0-a989-dd11fb78de5f" containerName="revoke-keys" Nov 02 22:12:12.805932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.805922 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b12d95-3e27-40d0-a989-dd11fb78de5f" containerName="revoke-keys" Nov 02 22:12:12.809701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.809653 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:12.866327 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.866293 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg"] Nov 02 22:12:12.897967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.897932 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxz2\" (UniqueName: \"kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:12.898137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.898112 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:12.898252 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.898228 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:12.909924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.909899 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:12.909924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.909909 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:12.911555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.911527 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:12:12.911555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.911531 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:12.999519 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.999487 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:12.999703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.999577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxz2\" (UniqueName: \"kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:12.999703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:12.999665 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:13.050277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.050248 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:13.050493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.050474 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:13.060858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.060804 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:13.062483 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.062462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:13.103603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.103580 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:13.116508 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.116474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxz2\" (UniqueName: \"kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:13.174253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.174221 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:12:13.183398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.183368 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:13.564712 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:12:13.564640 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bba61af_2721_435a_9c1f_bca33055f381.slice/crio-386667542d88c6eefa49b5c64405012fe7ae63aba7fc43ed6e5d62862b2f3d4c WatchSource:0}: Error finding container 386667542d88c6eefa49b5c64405012fe7ae63aba7fc43ed6e5d62862b2f3d4c: Status 404 returned error can't find the container with id 386667542d88c6eefa49b5c64405012fe7ae63aba7fc43ed6e5d62862b2f3d4c Nov 02 22:12:13.565067 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:13.565045 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg"] Nov 02 22:12:14.116512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:14.116476 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" event={"ID":"5bba61af-2721-435a-9c1f-bca33055f381","Type":"ContainerStarted","Data":"a9a5946a3a009d0f7528b64e4bd0bde3c9da4f85bb461903918c352c11ab2a17"} Nov 02 22:12:14.116909 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:14.116532 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" event={"ID":"5bba61af-2721-435a-9c1f-bca33055f381","Type":"ContainerStarted","Data":"386667542d88c6eefa49b5c64405012fe7ae63aba7fc43ed6e5d62862b2f3d4c"} Nov 02 22:12:14.264499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:14.264363 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" podStartSLOduration=2.2643397419999998 podStartE2EDuration="2.264339742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:14.236183096 +0000 UTC m=+2281.398278218" watchObservedRunningTime="2026-11-02 22:12:14.264339742 +0000 UTC m=+2281.426434863" Nov 02 22:12:14.701170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:14.701131 5091 patch_prober.go:29] interesting pod/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:14.701309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:14.701192 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:15.125403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:15.125371 5091 generic.go:356] "Generic (PLEG): container finished" podID="5bba61af-2721-435a-9c1f-bca33055f381" containerID="a9a5946a3a009d0f7528b64e4bd0bde3c9da4f85bb461903918c352c11ab2a17" exitCode=0 Nov 02 22:12:15.125867 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:15.125462 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" event={"ID":"5bba61af-2721-435a-9c1f-bca33055f381","Type":"ContainerDied","Data":"a9a5946a3a009d0f7528b64e4bd0bde3c9da4f85bb461903918c352c11ab2a17"} Nov 02 22:12:15.987818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:15.987761 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:12:16.136663 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.136630 5091 generic.go:356] "Generic (PLEG): container finished" podID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerID="38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c" exitCode=0 Nov 02 22:12:16.137166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.136744 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" Nov 02 22:12:16.137166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.136835 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" event={"ID":"2ae31de7-5263-4371-b452-ebb1b1edc4de","Type":"ContainerDied","Data":"38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c"} Nov 02 22:12:16.137166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.136900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv" event={"ID":"2ae31de7-5263-4371-b452-ebb1b1edc4de","Type":"ContainerDied","Data":"8e117cf901e47893a357ea4ce31413ba74966fd4e08ed39c833e6b7d718121f5"} Nov 02 22:12:16.137166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.136934 5091 scope.go:122] "RemoveContainer" containerID="38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c" Nov 02 22:12:16.141801 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.141773 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.141931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.141866 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.141999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.141935 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.141999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.141967 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.142109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142010 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.142109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142057 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.142219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142106 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.142280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142218 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:16.142280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142239 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.142396 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142281 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token\") pod \"2ae31de7-5263-4371-b452-ebb1b1edc4de\" (UID: \"2ae31de7-5263-4371-b452-ebb1b1edc4de\") " Nov 02 22:12:16.142460 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142406 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:16.142530 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142499 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:16.143128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.142962 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:16.143413 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.143391 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:16.144036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.143887 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.144036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.143917 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae31de7-5263-4371-b452-ebb1b1edc4de-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.144036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.143933 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.144036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.143949 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.144036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.143963 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.145112 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.145087 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:16.145813 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.145791 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:16.146144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.146116 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:16.146665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.146640 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8" pod "2ae31de7-5263-4371-b452-ebb1b1edc4de" (UID: "2ae31de7-5263-4371-b452-ebb1b1edc4de"). InnerVolumeSpecName "kube-api-access-drcg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:16.152712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.152559 5091 scope.go:122] "RemoveContainer" containerID="20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f" Nov 02 22:12:16.203507 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.203485 5091 scope.go:122] "RemoveContainer" containerID="38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c" Nov 02 22:12:16.203915 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:12:16.203893 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c\": container with ID starting with 38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c not found: ID does not exist" containerID="38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c" Nov 02 22:12:16.203995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.203924 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c"} err="failed to get container status \"38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c\": rpc error: code = NotFound desc = could not find container \"38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c\": container with ID starting with 38ed0973a22afd1f7dedbea635a083b107f470f498a59a939e3081535718f72c not found: ID does not exist" Nov 02 22:12:16.203995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.203943 5091 scope.go:122] "RemoveContainer" containerID="20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f" Nov 02 22:12:16.204269 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:12:16.204246 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f\": container with ID starting with 20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f not found: ID does not exist" containerID="20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f" Nov 02 22:12:16.204314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.204275 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f"} err="failed to get container status \"20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f\": rpc error: code = NotFound desc = could not find container \"20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f\": container with ID starting with 20467a8ce618d42342d6a0e51df9e674bf7201ad3d6fc16ef91ad6ffbbcc1e9f not found: ID does not exist" Nov 02 22:12:16.246079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.246007 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drcg8\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-kube-api-access-drcg8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.246079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.246044 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.246079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.246059 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.246079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.246071 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae31de7-5263-4371-b452-ebb1b1edc4de-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.302764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.302739 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:16.449058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.449033 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2\" (UniqueName: \"kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2\") pod \"5bba61af-2721-435a-9c1f-bca33055f381\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " Nov 02 22:12:16.449196 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.449183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca\") pod \"5bba61af-2721-435a-9c1f-bca33055f381\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " Nov 02 22:12:16.449325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.449305 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca\") pod \"5bba61af-2721-435a-9c1f-bca33055f381\" (UID: \"5bba61af-2721-435a-9c1f-bca33055f381\") " Nov 02 22:12:16.449740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.449700 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca" pod "5bba61af-2721-435a-9c1f-bca33055f381" (UID: "5bba61af-2721-435a-9c1f-bca33055f381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:16.449880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.449864 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.451441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.451415 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca" pod "5bba61af-2721-435a-9c1f-bca33055f381" (UID: "5bba61af-2721-435a-9c1f-bca33055f381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:16.451532 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.451424 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2" pod "5bba61af-2721-435a-9c1f-bca33055f381" (UID: "5bba61af-2721-435a-9c1f-bca33055f381"). InnerVolumeSpecName "kube-api-access-5nxz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:16.551302 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.551232 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxz2\" (UniqueName: \"kubernetes.io/projected/5bba61af-2721-435a-9c1f-bca33055f381-kube-api-access-5nxz2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.551302 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.551262 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bba61af-2721-435a-9c1f-bca33055f381-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:16.558876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.558849 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv"] Nov 02 22:12:16.564008 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.563984 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2050f3-openshift-default-6f6d6774f5-kgmkv"] Nov 02 22:12:16.977899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:16.977865 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" path="/var/lib/kubelet/pods/2ae31de7-5263-4371-b452-ebb1b1edc4de/volumes" Nov 02 22:12:17.146119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:17.146084 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" Nov 02 22:12:17.146469 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:17.146112 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg" event={"ID":"5bba61af-2721-435a-9c1f-bca33055f381","Type":"ContainerDied","Data":"386667542d88c6eefa49b5c64405012fe7ae63aba7fc43ed6e5d62862b2f3d4c"} Nov 02 22:12:17.146469 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:17.146163 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="386667542d88c6eefa49b5c64405012fe7ae63aba7fc43ed6e5d62862b2f3d4c" Nov 02 22:12:29.318888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:29.318850 5091 generic.go:356] "Generic (PLEG): container finished" podID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerID="2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61" exitCode=0 Nov 02 22:12:29.319439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:29.318928 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" event={"ID":"1bfc1424-b3de-491f-a94f-c9408f14b660","Type":"ContainerDied","Data":"2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61"} Nov 02 22:12:29.319439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:29.319369 5091 scope.go:122] "RemoveContainer" containerID="2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61" Nov 02 22:12:29.320920 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:29.320888 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:29.321042 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:29.320973 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:29.321042 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:29.321022 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:30.328051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.328015 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" event={"ID":"1bfc1424-b3de-491f-a94f-c9408f14b660","Type":"ContainerStarted","Data":"88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f"} Nov 02 22:12:30.534935 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.534901 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:12:30.535961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.535942 5091 patch_prober.go:29] interesting pod/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 22:12:30.536020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.535994 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 22:12:30.692395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.692360 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c"] Nov 02 22:12:30.692655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.692629 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" containerID="cri-o://76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51" gracePeriod=30 Nov 02 22:12:30.700398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:30.700368 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:12:31.534896 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:31.534860 5091 patch_prober.go:29] interesting pod/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 22:12:31.535283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:31.534906 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 22:12:32.031580 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.031544 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5"] Nov 02 22:12:32.039277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.037801 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd"] Nov 02 22:12:32.039277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.038010 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" containerID="cri-o://44bef8d4d8e358829f200a8e86345e77eae4c6f5b813c0fa384414f2fa7dea17" gracePeriod=30 Nov 02 22:12:32.039277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.038271 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" containerID="cri-o://d5b296797ce8c6b1b24191e21cc87e75a3451ae1e9e90c1e4225ef0294ef743c" gracePeriod=30 Nov 02 22:12:32.068734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.055380 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g"] Nov 02 22:12:32.086657 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.086606 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" containerID="cri-o://429780485c77c638766ce9f824d89fdd6406adfc9f8e5f8815ad1df357487ebe" gracePeriod=30 Nov 02 22:12:32.535160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.535124 5091 patch_prober.go:29] interesting pod/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 22:12:32.535569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:32.535178 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 22:12:33.534641 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:33.534602 5091 patch_prober.go:29] interesting pod/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 22:12:33.534816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:33.534666 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 22:12:34.124143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.124109 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:34.124544 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.124176 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:34.534504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.534408 5091 patch_prober.go:29] interesting pod/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 22:12:34.534504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.534473 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 22:12:34.890233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.890186 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg"] Nov 02 22:12:34.930354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.930316 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d1c62e-n7qrg"] Nov 02 22:12:34.994529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:34.994496 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bba61af-2721-435a-9c1f-bca33055f381" path="/var/lib/kubelet/pods/5bba61af-2721-435a-9c1f-bca33055f381/volumes" Nov 02 22:12:35.200736 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.196981 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:35.200736 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.197041 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:35.303736 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.303700 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:35.303906 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.303768 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:35.318120 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.318090 5091 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:35.318264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.318136 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:35.534244 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.534162 5091 patch_prober.go:29] interesting pod/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 22:12:35.534244 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.534214 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 22:12:35.975215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:35.975184 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 22:12:36.060819 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.060779 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.060970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.060896 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.060970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.060932 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.060986 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061090 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061162 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:36.061279 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061334 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061306 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061366 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token\") pod \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\" (UID: \"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95\") " Nov 02 22:12:36.061401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061365 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:36.061523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:36.061523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061477 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:36.061631 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.061611 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:36.062323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.062260 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.062323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.062289 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.062323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.062306 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.062323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.062320 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.062611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.062337 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.063375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.063348 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:36.063654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.063634 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:36.063927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.063909 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "kube-api-access-xnlb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:36.064748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.064730 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy" pod "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" (UID: "48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:36.163837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.163794 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.163837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.163836 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlb8\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-kube-api-access-xnlb8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.164035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.163855 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.164035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.163870 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.392772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.392741 5091 generic.go:356] "Generic (PLEG): container finished" podID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerID="76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51" exitCode=0 Nov 02 22:12:36.393233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.392798 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" event={"ID":"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95","Type":"ContainerDied","Data":"76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51"} Nov 02 22:12:36.393233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.392835 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" event={"ID":"48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95","Type":"ContainerDied","Data":"b01e71aa29a654b96dfbf39291d1e3ea883eef61282b9a105611335a145ffd61"} Nov 02 22:12:36.393233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.392856 5091 scope.go:122] "RemoveContainer" containerID="76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51" Nov 02 22:12:36.393233 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.392801 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c" Nov 02 22:12:36.414157 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.414133 5091 scope.go:122] "RemoveContainer" containerID="76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51" Nov 02 22:12:36.414476 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:12:36.414453 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51\": container with ID starting with 76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51 not found: ID does not exist" containerID="76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51" Nov 02 22:12:36.414533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.414488 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51"} err="failed to get container status \"76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51\": rpc error: code = NotFound desc = could not find container \"76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51\": container with ID starting with 76f21b2292ac0c2db073431236963e3145397657ca5c35f93594d55c17867d51 not found: ID does not exist" Nov 02 22:12:36.538920 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.538892 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:12:36.539760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.539740 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:12:36.557966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.557939 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c"] Nov 02 22:12:36.607460 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.607433 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-scd2c"] Nov 02 22:12:36.977494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:36.977452 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" path="/var/lib/kubelet/pods/48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95/volumes" Nov 02 22:12:37.121029 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:12:37.120907 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:12:37.415354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.415316 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerID="d5b296797ce8c6b1b24191e21cc87e75a3451ae1e9e90c1e4225ef0294ef743c" exitCode=0 Nov 02 22:12:37.415862 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.415431 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" event={"ID":"c1e86e73-5891-4885-a3ed-e7ea709cddd6","Type":"ContainerDied","Data":"d5b296797ce8c6b1b24191e21cc87e75a3451ae1e9e90c1e4225ef0294ef743c"} Nov 02 22:12:37.435836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.432565 5091 generic.go:356] "Generic (PLEG): container finished" podID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerID="44bef8d4d8e358829f200a8e86345e77eae4c6f5b813c0fa384414f2fa7dea17" exitCode=0 Nov 02 22:12:37.435836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.432899 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" event={"ID":"acf86efb-606d-4fc4-9b97-b7697e2216d2","Type":"ContainerDied","Data":"44bef8d4d8e358829f200a8e86345e77eae4c6f5b813c0fa384414f2fa7dea17"} Nov 02 22:12:37.435836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.435303 5091 generic.go:356] "Generic (PLEG): container finished" podID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerID="429780485c77c638766ce9f824d89fdd6406adfc9f8e5f8815ad1df357487ebe" exitCode=0 Nov 02 22:12:37.436122 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.436094 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" event={"ID":"91c36ca8-9b66-4235-8531-4e7e1343a475","Type":"ContainerDied","Data":"429780485c77c638766ce9f824d89fdd6406adfc9f8e5f8815ad1df357487ebe"} Nov 02 22:12:37.476781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.476734 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:12:37.496071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496113 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496168 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496220 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496351 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496383 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.496555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.496521 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2\") pod \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\" (UID: \"c1e86e73-5891-4885-a3ed-e7ea709cddd6\") " Nov 02 22:12:37.497927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.497861 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.498414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.498262 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.498981 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.498536 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.499083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.499027 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.499295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.499233 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.501958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.501926 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:37.503332 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.503300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.503440 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.503379 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "kube-api-access-nmzr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.504417 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.504393 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token" pod "c1e86e73-5891-4885-a3ed-e7ea709cddd6" (UID: "c1e86e73-5891-4885-a3ed-e7ea709cddd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.540950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.540922 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:12:37.592472 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.592447 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:12:37.598114 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598085 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.598280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598137 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.598280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598168 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.598411 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598338 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.598411 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598398 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.598603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598493 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.598715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.598644 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.599189 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599162 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.599318 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599254 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.599318 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599305 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599340 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599378 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.599435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599416 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.599435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599428 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599455 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.599653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599499 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599593 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599648 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.599956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599699 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.599956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599724 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.599956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599731 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy\") pod \"91c36ca8-9b66-4235-8531-4e7e1343a475\" (UID: \"91c36ca8-9b66-4235-8531-4e7e1343a475\") " Nov 02 22:12:37.599956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.599811 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk\") pod \"acf86efb-606d-4fc4-9b97-b7697e2216d2\" (UID: \"acf86efb-606d-4fc4-9b97-b7697e2216d2\") " Nov 02 22:12:37.600249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.600224 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.601006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.600449 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.601006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.600663 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.601006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.600862 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:37.601297 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.601194 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.601456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.601431 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.602438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.602404 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.603594 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.603563 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.604393 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.604370 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:37.605628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605597 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "kube-api-access-dfggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.605915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605892 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.605963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605923 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.605963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605938 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzr2\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-kube-api-access-nmzr2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.605963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605952 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605965 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c36ca8-9b66-4235-8531-4e7e1343a475-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605978 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.605993 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606006 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606019 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606032 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606045 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf86efb-606d-4fc4-9b97-b7697e2216d2-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606058 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606071 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606084 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606096 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfggk\" (UniqueName: \"kubernetes.io/projected/acf86efb-606d-4fc4-9b97-b7697e2216d2-kube-api-access-dfggk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606109 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606124 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606139 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606152 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606167 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606180 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606193 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c36ca8-9b66-4235-8531-4e7e1343a475-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606207 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e86e73-5891-4885-a3ed-e7ea709cddd6-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.606383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.606221 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e86e73-5891-4885-a3ed-e7ea709cddd6-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.608659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.608638 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy" pod "acf86efb-606d-4fc4-9b97-b7697e2216d2" (UID: "acf86efb-606d-4fc4-9b97-b7697e2216d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.609224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.609202 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "kube-api-access-hzgfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.611942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.611917 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token" pod "91c36ca8-9b66-4235-8531-4e7e1343a475" (UID: "91c36ca8-9b66-4235-8531-4e7e1343a475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.707003 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.706975 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.707003 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.707009 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgfm\" (UniqueName: \"kubernetes.io/projected/91c36ca8-9b66-4235-8531-4e7e1343a475-kube-api-access-hzgfm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.707207 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:37.707019 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf86efb-606d-4fc4-9b97-b7697e2216d2-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:38.444095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.444065 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" Nov 02 22:12:38.444095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.444091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5" event={"ID":"c1e86e73-5891-4885-a3ed-e7ea709cddd6","Type":"ContainerDied","Data":"64fcd6c86df4d7db4a29641d141edb1fc725ba24786efb3e519fe851c086fc71"} Nov 02 22:12:38.444657 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.444152 5091 scope.go:122] "RemoveContainer" containerID="d5b296797ce8c6b1b24191e21cc87e75a3451ae1e9e90c1e4225ef0294ef743c" Nov 02 22:12:38.445773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.445714 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" Nov 02 22:12:38.445891 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.445715 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd" event={"ID":"acf86efb-606d-4fc4-9b97-b7697e2216d2","Type":"ContainerDied","Data":"a7e41b7ebce2fd0d01ed67cdfefcf4c169ec054e7ba9d7341df6e27f0a4e234d"} Nov 02 22:12:38.447214 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.447192 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" Nov 02 22:12:38.447317 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.447215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g" event={"ID":"91c36ca8-9b66-4235-8531-4e7e1343a475","Type":"ContainerDied","Data":"54914e2ab2abaf71ac0c89c0b39ff19145574bc6485d1332b3a78f5e82d7548a"} Nov 02 22:12:38.455802 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.455762 5091 scope.go:122] "RemoveContainer" containerID="44bef8d4d8e358829f200a8e86345e77eae4c6f5b813c0fa384414f2fa7dea17" Nov 02 22:12:38.466484 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.466466 5091 scope.go:122] "RemoveContainer" containerID="429780485c77c638766ce9f824d89fdd6406adfc9f8e5f8815ad1df357487ebe" Nov 02 22:12:38.609869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.609837 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5"] Nov 02 22:12:38.658142 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.658107 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-crxn5"] Nov 02 22:12:38.720303 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.720220 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g"] Nov 02 22:12:38.724811 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.724783 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5v58g"] Nov 02 22:12:38.839539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.839502 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd"] Nov 02 22:12:38.840154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.840133 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-6rlgd"] Nov 02 22:12:38.977985 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.977908 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" path="/var/lib/kubelet/pods/91c36ca8-9b66-4235-8531-4e7e1343a475/volumes" Nov 02 22:12:38.978433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.978420 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" path="/var/lib/kubelet/pods/acf86efb-606d-4fc4-9b97-b7697e2216d2/volumes" Nov 02 22:12:38.978909 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:38.978895 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" path="/var/lib/kubelet/pods/c1e86e73-5891-4885-a3ed-e7ea709cddd6/volumes" Nov 02 22:12:40.028292 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.028217 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz"] Nov 02 22:12:40.031050 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.031026 5091 cpu_manager.go:414] "containerMap: removing container" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" Nov 02 22:12:40.031230 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.031217 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" Nov 02 22:12:40.031849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.031832 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" Nov 02 22:12:40.031958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.031947 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" Nov 02 22:12:40.032051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032041 5091 cpu_manager.go:414] "containerMap: removing container" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" Nov 02 22:12:40.032128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032118 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" Nov 02 22:12:40.032231 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032205 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" Nov 02 22:12:40.032231 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032222 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" Nov 02 22:12:40.032423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032249 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5bba61af-2721-435a-9c1f-bca33055f381" containerName="revoke-keys" Nov 02 22:12:40.032509 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032490 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bba61af-2721-435a-9c1f-bca33055f381" containerName="revoke-keys" Nov 02 22:12:40.032571 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032519 5091 cpu_manager.go:414] "containerMap: removing container" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" Nov 02 22:12:40.032571 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032532 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" Nov 02 22:12:40.032731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032588 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" Nov 02 22:12:40.032731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.032596 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" Nov 02 22:12:40.034727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034702 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bba61af-2721-435a-9c1f-bca33055f381" containerName="revoke-keys" Nov 02 22:12:40.034847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034732 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="48cac4d4-2f5b-4ac4-b171-eae2bf7b6e95" containerName="istio-proxy" Nov 02 22:12:40.034847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034749 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e86e73-5891-4885-a3ed-e7ea709cddd6" containerName="istio-proxy" Nov 02 22:12:40.034847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034761 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c36ca8-9b66-4235-8531-4e7e1343a475" containerName="istio-proxy" Nov 02 22:12:40.034847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034778 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" Nov 02 22:12:40.034847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034788 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf86efb-606d-4fc4-9b97-b7697e2216d2" containerName="istio-proxy" Nov 02 22:12:40.034847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.034799 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae31de7-5263-4371-b452-ebb1b1edc4de" containerName="istio-proxy" Nov 02 22:12:40.055012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.054977 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.089358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.089318 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz"] Nov 02 22:12:40.133799 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.133767 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:40.133986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.133817 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:12:40.133986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.133839 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:40.134154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.134127 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:40.151739 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.151714 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.151853 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.151834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zlp\" (UniqueName: \"kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.151898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.151869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.253136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.253107 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.253271 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.253170 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zlp\" (UniqueName: \"kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.253348 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.253327 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.253796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.253774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.255698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.255665 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.310140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.310081 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zlp\" (UniqueName: \"kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp\") pod \"maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.380236 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.380202 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:40.760080 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:40.760057 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz"] Nov 02 22:12:40.763106 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:12:40.763073 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3aa39f_91e1_4af6_af2a_0b5c3286de4f.slice/crio-ea61b5a0c5cf71b31eb123b5f49a11168f43d035f2ea02b726ff7ef9c0edc526 WatchSource:0}: Error finding container ea61b5a0c5cf71b31eb123b5f49a11168f43d035f2ea02b726ff7ef9c0edc526: Status 404 returned error can't find the container with id ea61b5a0c5cf71b31eb123b5f49a11168f43d035f2ea02b726ff7ef9c0edc526 Nov 02 22:12:41.287210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:41.287178 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs"] Nov 02 22:12:41.342439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:41.342394 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587629-vmshs"] Nov 02 22:12:41.496499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:41.496398 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" event={"ID":"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f","Type":"ContainerStarted","Data":"afd3d630d4d6596197c8f31439545e52cdeee0ba395cfb4e3a6dce45faed6d9d"} Nov 02 22:12:41.496499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:41.496451 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" event={"ID":"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f","Type":"ContainerStarted","Data":"ea61b5a0c5cf71b31eb123b5f49a11168f43d035f2ea02b726ff7ef9c0edc526"} Nov 02 22:12:41.611550 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:41.611487 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" podStartSLOduration=2.6114718630000002 podStartE2EDuration="2.611471863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:41.609367719 +0000 UTC m=+2308.771462858" watchObservedRunningTime="2026-11-02 22:12:41.611471863 +0000 UTC m=+2308.773566982" Nov 02 22:12:42.393823 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.393772 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj"] Nov 02 22:12:42.397942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.397924 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.454874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.454836 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj"] Nov 02 22:12:42.498216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.498178 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64mj\" (UniqueName: \"kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.498402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.498385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.498453 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.498439 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.505989 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.505962 5091 generic.go:356] "Generic (PLEG): container finished" podID="ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" containerID="afd3d630d4d6596197c8f31439545e52cdeee0ba395cfb4e3a6dce45faed6d9d" exitCode=0 Nov 02 22:12:42.506126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.506042 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" event={"ID":"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f","Type":"ContainerDied","Data":"afd3d630d4d6596197c8f31439545e52cdeee0ba395cfb4e3a6dce45faed6d9d"} Nov 02 22:12:42.599502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.599460 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64mj\" (UniqueName: \"kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.599707 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.599625 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.599808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.599723 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.600435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.600416 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.602099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.602077 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.658021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.657945 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64mj\" (UniqueName: \"kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.713352 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.713322 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:42.978594 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:42.978515 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff92398-fa9c-4df1-ab68-a33649d3d2f8" path="/var/lib/kubelet/pods/1ff92398-fa9c-4df1-ab68-a33649d3d2f8/volumes" Nov 02 22:12:43.096728 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.093922 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj"] Nov 02 22:12:43.099873 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:12:43.099822 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49b1b39_a659_4a94_9305_bdd50ce239fb.slice/crio-39f37cab9c5ca75e3456c90684d417ede3a91212ac3a78879ccfb3213bb02610 WatchSource:0}: Error finding container 39f37cab9c5ca75e3456c90684d417ede3a91212ac3a78879ccfb3213bb02610: Status 404 returned error can't find the container with id 39f37cab9c5ca75e3456c90684d417ede3a91212ac3a78879ccfb3213bb02610 Nov 02 22:12:43.515523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.515480 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" event={"ID":"e49b1b39-a659-4a94-9305-bdd50ce239fb","Type":"ContainerStarted","Data":"ffb00204ee22650f659706b5c72276a3d1c9bedf7ebeb7597bcff2843adeb08b"} Nov 02 22:12:43.515993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.515547 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" event={"ID":"e49b1b39-a659-4a94-9305-bdd50ce239fb","Type":"ContainerStarted","Data":"39f37cab9c5ca75e3456c90684d417ede3a91212ac3a78879ccfb3213bb02610"} Nov 02 22:12:43.630879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.630826 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" podStartSLOduration=1.6308113359999998 podStartE2EDuration="1.630811336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:43.628544642 +0000 UTC m=+2310.790639764" watchObservedRunningTime="2026-11-02 22:12:43.630811336 +0000 UTC m=+2310.792906455" Nov 02 22:12:43.675034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.675003 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:43.816688 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.816641 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca\") pod \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " Nov 02 22:12:43.816899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.816877 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp\" (UniqueName: \"kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp\") pod \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " Nov 02 22:12:43.816975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.816925 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca\") pod \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\" (UID: \"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f\") " Nov 02 22:12:43.817095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.817072 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca" pod "ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" (UID: "ac3aa39f-91e1-4af6-af2a-0b5c3286de4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:43.817541 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.817514 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:43.819247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.819222 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca" pod "ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" (UID: "ac3aa39f-91e1-4af6-af2a-0b5c3286de4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:43.819778 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.819749 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp" pod "ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" (UID: "ac3aa39f-91e1-4af6-af2a-0b5c3286de4f"). InnerVolumeSpecName "kube-api-access-x8zlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:43.919077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.919046 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zlp\" (UniqueName: \"kubernetes.io/projected/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-kube-api-access-x8zlp\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:43.919077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:43.919074 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:44.524884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:44.524852 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" Nov 02 22:12:44.525291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:44.524890 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz" event={"ID":"ac3aa39f-91e1-4af6-af2a-0b5c3286de4f","Type":"ContainerDied","Data":"ea61b5a0c5cf71b31eb123b5f49a11168f43d035f2ea02b726ff7ef9c0edc526"} Nov 02 22:12:44.525291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:44.524953 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea61b5a0c5cf71b31eb123b5f49a11168f43d035f2ea02b726ff7ef9c0edc526" Nov 02 22:12:44.526915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:44.526885 5091 generic.go:356] "Generic (PLEG): container finished" podID="e49b1b39-a659-4a94-9305-bdd50ce239fb" containerID="ffb00204ee22650f659706b5c72276a3d1c9bedf7ebeb7597bcff2843adeb08b" exitCode=0 Nov 02 22:12:44.527084 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:44.526971 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" event={"ID":"e49b1b39-a659-4a94-9305-bdd50ce239fb","Type":"ContainerDied","Data":"ffb00204ee22650f659706b5c72276a3d1c9bedf7ebeb7597bcff2843adeb08b"} Nov 02 22:12:45.687259 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.687232 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:45.851326 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.851246 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj\" (UniqueName: \"kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj\") pod \"e49b1b39-a659-4a94-9305-bdd50ce239fb\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " Nov 02 22:12:45.851471 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.851352 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca\") pod \"e49b1b39-a659-4a94-9305-bdd50ce239fb\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " Nov 02 22:12:45.851600 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.851582 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca\") pod \"e49b1b39-a659-4a94-9305-bdd50ce239fb\" (UID: \"e49b1b39-a659-4a94-9305-bdd50ce239fb\") " Nov 02 22:12:45.852026 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.852001 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca" pod "e49b1b39-a659-4a94-9305-bdd50ce239fb" (UID: "e49b1b39-a659-4a94-9305-bdd50ce239fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:45.852155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.852139 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:45.854014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.853986 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj" pod "e49b1b39-a659-4a94-9305-bdd50ce239fb" (UID: "e49b1b39-a659-4a94-9305-bdd50ce239fb"). InnerVolumeSpecName "kube-api-access-b64mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:45.854249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.854228 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca" pod "e49b1b39-a659-4a94-9305-bdd50ce239fb" (UID: "e49b1b39-a659-4a94-9305-bdd50ce239fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:45.953011 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.952970 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64mj\" (UniqueName: \"kubernetes.io/projected/e49b1b39-a659-4a94-9305-bdd50ce239fb-kube-api-access-b64mj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:45.953011 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:45.953006 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49b1b39-a659-4a94-9305-bdd50ce239fb-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:46.553143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:46.553107 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" Nov 02 22:12:46.553323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:46.553142 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj" event={"ID":"e49b1b39-a659-4a94-9305-bdd50ce239fb","Type":"ContainerDied","Data":"39f37cab9c5ca75e3456c90684d417ede3a91212ac3a78879ccfb3213bb02610"} Nov 02 22:12:46.553323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:46.553213 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f37cab9c5ca75e3456c90684d417ede3a91212ac3a78879ccfb3213bb02610" Nov 02 22:12:47.200941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:47.200900 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2"] Nov 02 22:12:47.201453 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:47.201147 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" containerID="cri-o://c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4" gracePeriod=30 Nov 02 22:12:49.047352 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.047320 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x"] Nov 02 22:12:49.048134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.048114 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" containerName="revoke-keys" Nov 02 22:12:49.048134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.048135 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" containerName="revoke-keys" Nov 02 22:12:49.048281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.048144 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e49b1b39-a659-4a94-9305-bdd50ce239fb" containerName="revoke-keys" Nov 02 22:12:49.048281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.048152 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49b1b39-a659-4a94-9305-bdd50ce239fb" containerName="revoke-keys" Nov 02 22:12:49.048281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.048273 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" containerName="revoke-keys" Nov 02 22:12:49.048451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.048284 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49b1b39-a659-4a94-9305-bdd50ce239fb" containerName="revoke-keys" Nov 02 22:12:49.052591 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.052570 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.109203 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.109171 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x"] Nov 02 22:12:49.154016 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.153986 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:49.154199 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.154035 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:12:49.154199 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.154070 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:49.201609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.201576 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.201813 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.201758 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.201879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.201858 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6mk\" (UniqueName: \"kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.206426 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.206408 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:49.303009 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.302925 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6mk\" (UniqueName: \"kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.303160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.303084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.303214 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.303199 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.303739 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.303717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.305638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.305608 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.359361 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.359327 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6mk\" (UniqueName: \"kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk\") pod \"maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.366793 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.366768 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:49.747148 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:49.747122 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x"] Nov 02 22:12:49.749395 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:12:49.749361 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8137518d_9f8a_4f9f_90e1_565422e58f87.slice/crio-c317ced2d9098165922f0dde58427b4c7b99f8187b5f08de03480efafb9f97af WatchSource:0}: Error finding container c317ced2d9098165922f0dde58427b4c7b99f8187b5f08de03480efafb9f97af: Status 404 returned error can't find the container with id c317ced2d9098165922f0dde58427b4c7b99f8187b5f08de03480efafb9f97af Nov 02 22:12:50.614550 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:50.614510 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" event={"ID":"8137518d-9f8a-4f9f-90e1-565422e58f87","Type":"ContainerStarted","Data":"e7d8ceff5ec2b24c990a89d67fef9ecc0335cfe78c9dd0d03784a2f80d05508a"} Nov 02 22:12:50.615038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:50.614596 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" event={"ID":"8137518d-9f8a-4f9f-90e1-565422e58f87","Type":"ContainerStarted","Data":"c317ced2d9098165922f0dde58427b4c7b99f8187b5f08de03480efafb9f97af"} Nov 02 22:12:50.727791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:50.727740 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" podStartSLOduration=1.7277256140000001 podStartE2EDuration="1.727725614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:50.725596862 +0000 UTC m=+2317.887691983" watchObservedRunningTime="2026-11-02 22:12:50.727725614 +0000 UTC m=+2317.889820733" Nov 02 22:12:51.410186 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:51.410149 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:51.410340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:51.410204 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:51.623556 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:51.623524 5091 generic.go:356] "Generic (PLEG): container finished" podID="8137518d-9f8a-4f9f-90e1-565422e58f87" containerID="e7d8ceff5ec2b24c990a89d67fef9ecc0335cfe78c9dd0d03784a2f80d05508a" exitCode=0 Nov 02 22:12:51.623996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:51.623615 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" event={"ID":"8137518d-9f8a-4f9f-90e1-565422e58f87","Type":"ContainerDied","Data":"e7d8ceff5ec2b24c990a89d67fef9ecc0335cfe78c9dd0d03784a2f80d05508a"} Nov 02 22:12:52.477242 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.477182 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 22:12:52.556462 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556429 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556612 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556473 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556612 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556528 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556623 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556742 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556934 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556801 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556934 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556894 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.556934 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.556930 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token\") pod \"da17aa08-9009-40bc-8576-643e3d832fe4\" (UID: \"da17aa08-9009-40bc-8576-643e3d832fe4\") " Nov 02 22:12:52.557143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557109 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:52.557143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557142 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:52.557345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557156 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:52.557518 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557489 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:52.557609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557575 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:52.557710 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557663 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.557790 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557713 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.557790 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557728 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.557790 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.557744 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.559228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.559204 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:52.559330 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.559251 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "kube-api-access-hbcrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:52.559330 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.559291 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:52.559425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.559320 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo" pod "da17aa08-9009-40bc-8576-643e3d832fe4" (UID: "da17aa08-9009-40bc-8576-643e3d832fe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:52.632605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.632569 5091 generic.go:356] "Generic (PLEG): container finished" podID="da17aa08-9009-40bc-8576-643e3d832fe4" containerID="c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4" exitCode=0 Nov 02 22:12:52.633088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.632626 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" Nov 02 22:12:52.633088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.632659 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" event={"ID":"da17aa08-9009-40bc-8576-643e3d832fe4","Type":"ContainerDied","Data":"c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4"} Nov 02 22:12:52.633088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.632751 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2" event={"ID":"da17aa08-9009-40bc-8576-643e3d832fe4","Type":"ContainerDied","Data":"5d7c569ff55a2ef3e429a2e79176df0c0a4f6a79fd49f9339d9d206dfd2457ec"} Nov 02 22:12:52.633088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.632774 5091 scope.go:122] "RemoveContainer" containerID="c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4" Nov 02 22:12:52.647201 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.646664 5091 scope.go:122] "RemoveContainer" containerID="c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4" Nov 02 22:12:52.647833 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:12:52.647802 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4\": container with ID starting with c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4 not found: ID does not exist" containerID="c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4" Nov 02 22:12:52.647944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.647844 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4"} err="failed to get container status \"c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4\": rpc error: code = NotFound desc = could not find container \"c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4\": container with ID starting with c0902d75a493756f756fb6fea594d7ac2e67a63768553010b5104e8c5e45ffd4 not found: ID does not exist" Nov 02 22:12:52.659350 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.659311 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da17aa08-9009-40bc-8576-643e3d832fe4-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.659350 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.659352 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da17aa08-9009-40bc-8576-643e3d832fe4-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.659531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.659363 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.659531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.659372 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da17aa08-9009-40bc-8576-643e3d832fe4-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.659531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.659381 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcrv\" (UniqueName: \"kubernetes.io/projected/da17aa08-9009-40bc-8576-643e3d832fe4-kube-api-access-hbcrv\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.782454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.782424 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:52.839372 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.839337 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2"] Nov 02 22:12:52.842929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.842902 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-8lxj2"] Nov 02 22:12:52.863638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.863609 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk\" (UniqueName: \"kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk\") pod \"8137518d-9f8a-4f9f-90e1-565422e58f87\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " Nov 02 22:12:52.863767 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.863740 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca\") pod \"8137518d-9f8a-4f9f-90e1-565422e58f87\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " Nov 02 22:12:52.863827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.863788 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca\") pod \"8137518d-9f8a-4f9f-90e1-565422e58f87\" (UID: \"8137518d-9f8a-4f9f-90e1-565422e58f87\") " Nov 02 22:12:52.864095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.864068 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca" pod "8137518d-9f8a-4f9f-90e1-565422e58f87" (UID: "8137518d-9f8a-4f9f-90e1-565422e58f87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:52.864292 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.864275 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.865890 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.865870 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk" pod "8137518d-9f8a-4f9f-90e1-565422e58f87" (UID: "8137518d-9f8a-4f9f-90e1-565422e58f87"). InnerVolumeSpecName "kube-api-access-hh6mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:52.865890 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.865878 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca" pod "8137518d-9f8a-4f9f-90e1-565422e58f87" (UID: "8137518d-9f8a-4f9f-90e1-565422e58f87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:52.978397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.973187 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh6mk\" (UniqueName: \"kubernetes.io/projected/8137518d-9f8a-4f9f-90e1-565422e58f87-kube-api-access-hh6mk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.978397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.973227 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8137518d-9f8a-4f9f-90e1-565422e58f87-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.986531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:52.986498 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" path="/var/lib/kubelet/pods/da17aa08-9009-40bc-8576-643e3d832fe4/volumes" Nov 02 22:12:53.642080 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:53.642048 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" Nov 02 22:12:53.642564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:53.642084 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x" event={"ID":"8137518d-9f8a-4f9f-90e1-565422e58f87","Type":"ContainerDied","Data":"c317ced2d9098165922f0dde58427b4c7b99f8187b5f08de03480efafb9f97af"} Nov 02 22:12:53.642564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:12:53.642137 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c317ced2d9098165922f0dde58427b4c7b99f8187b5f08de03480efafb9f97af" Nov 02 22:13:07.261707 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:13:07.260893 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:13:07.436695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:07.420800 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp"] Nov 02 22:13:07.451962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:07.451913 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17435d-nw6gp"] Nov 02 22:13:08.978074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:08.978044 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb" path="/var/lib/kubelet/pods/bad4a9a1-b65e-4cdb-b797-da7ece8ba4bb/volumes" Nov 02 22:13:11.848520 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.848482 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst"] Nov 02 22:13:11.849088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.849073 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8137518d-9f8a-4f9f-90e1-565422e58f87" containerName="revoke-keys" Nov 02 22:13:11.849139 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.849088 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8137518d-9f8a-4f9f-90e1-565422e58f87" containerName="revoke-keys" Nov 02 22:13:11.849175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.849159 5091 cpu_manager.go:414] "containerMap: removing container" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" Nov 02 22:13:11.849175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.849170 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" Nov 02 22:13:11.849316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.849304 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8137518d-9f8a-4f9f-90e1-565422e58f87" containerName="revoke-keys" Nov 02 22:13:11.849359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.849327 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="da17aa08-9009-40bc-8576-643e3d832fe4" containerName="istio-proxy" Nov 02 22:13:11.853510 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.853486 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:11.902058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.902024 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms"] Nov 02 22:13:11.906474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.906449 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:11.909363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.909337 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst"] Nov 02 22:13:11.915284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.915261 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:11.915429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.915306 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8k6\" (UniqueName: \"kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:11.915429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.915376 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:11.947856 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.947828 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:11.954156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.954129 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:11.954156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.954138 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:13:11.954381 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:11.954173 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:12.011790 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.011757 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms"] Nov 02 22:13:12.016833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.016804 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8k6\" (UniqueName: \"kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.017054 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.016935 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.017054 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.017016 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.017054 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.017050 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2bx\" (UniqueName: \"kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.017247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.017103 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.017452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.017424 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.018877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.018849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.023091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.023066 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.073338 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.073305 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8k6\" (UniqueName: \"kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6\") pod \"maas-api-revoke-sub-aa7c62b3-7mhst\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.119478 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.119445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.119478 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.119489 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2bx\" (UniqueName: \"kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.119795 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.119535 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.120465 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.120431 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.122069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.122047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.168372 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.168340 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:12.177866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.177839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2bx\" (UniqueName: \"kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx\") pod \"maas-api-revoke-sub-af481bea-9fhms\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.227383 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.227349 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:12.554346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.554314 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst"] Nov 02 22:13:12.554956 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:13:12.554928 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44628257_a764_488d_9128_8a86fa97d47c.slice/crio-438d0817f64b6a2e651dd67467eeaeb1dc7ff9ef7d1d5abae42f8237ad54e89a WatchSource:0}: Error finding container 438d0817f64b6a2e651dd67467eeaeb1dc7ff9ef7d1d5abae42f8237ad54e89a: Status 404 returned error can't find the container with id 438d0817f64b6a2e651dd67467eeaeb1dc7ff9ef7d1d5abae42f8237ad54e89a Nov 02 22:13:12.662038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.662005 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms"] Nov 02 22:13:12.662401 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:13:12.662372 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c163b3_f8a5_485e_b316_abaf22ebdef9.slice/crio-5cadd299f39b5c941c2123b9c8aa87d8af5e3b0c21d393c182bd63763fc89057 WatchSource:0}: Error finding container 5cadd299f39b5c941c2123b9c8aa87d8af5e3b0c21d393c182bd63763fc89057: Status 404 returned error can't find the container with id 5cadd299f39b5c941c2123b9c8aa87d8af5e3b0c21d393c182bd63763fc89057 Nov 02 22:13:12.879152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.879110 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" event={"ID":"44628257-a764-488d-9128-8a86fa97d47c","Type":"ContainerStarted","Data":"438d0817f64b6a2e651dd67467eeaeb1dc7ff9ef7d1d5abae42f8237ad54e89a"} Nov 02 22:13:12.880763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:12.880737 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" event={"ID":"24c163b3-f8a5-485e-b316-abaf22ebdef9","Type":"ContainerStarted","Data":"5cadd299f39b5c941c2123b9c8aa87d8af5e3b0c21d393c182bd63763fc89057"} Nov 02 22:13:13.688628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:13.688597 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9"] Nov 02 22:13:13.688955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:13.688927 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" containerID="cri-o://43e19846384ce5cda96733c6ff5dbde027f6162a9380aa18610c350ab1ea74c5" gracePeriod=30 Nov 02 22:13:13.903928 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:13.902266 5091 generic.go:356] "Generic (PLEG): container finished" podID="24c163b3-f8a5-485e-b316-abaf22ebdef9" containerID="5559cd8c088b6cb12a92e07c1584c43a51d0081c2f47e6db4e85052d4aa3f695" exitCode=0 Nov 02 22:13:13.903928 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:13.902574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" event={"ID":"24c163b3-f8a5-485e-b316-abaf22ebdef9","Type":"ContainerDied","Data":"5559cd8c088b6cb12a92e07c1584c43a51d0081c2f47e6db4e85052d4aa3f695"} Nov 02 22:13:13.906906 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:13.906875 5091 generic.go:356] "Generic (PLEG): container finished" podID="44628257-a764-488d-9128-8a86fa97d47c" containerID="4c4311bb75d570b4b57721411b0e97cd77238ab0189eeedc69c89cd93ea8e571" exitCode=0 Nov 02 22:13:13.907037 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:13.906934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" event={"ID":"44628257-a764-488d-9128-8a86fa97d47c","Type":"ContainerDied","Data":"4c4311bb75d570b4b57721411b0e97cd77238ab0189eeedc69c89cd93ea8e571"} Nov 02 22:13:15.116879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.116856 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:15.120176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.120155 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:15.189616 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.189536 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca\") pod \"44628257-a764-488d-9128-8a86fa97d47c\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " Nov 02 22:13:15.189766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.189614 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6\" (UniqueName: \"kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6\") pod \"44628257-a764-488d-9128-8a86fa97d47c\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " Nov 02 22:13:15.189766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.189682 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca\") pod \"44628257-a764-488d-9128-8a86fa97d47c\" (UID: \"44628257-a764-488d-9128-8a86fa97d47c\") " Nov 02 22:13:15.189905 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.189830 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca\") pod \"24c163b3-f8a5-485e-b316-abaf22ebdef9\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " Nov 02 22:13:15.189905 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.189878 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca\") pod \"24c163b3-f8a5-485e-b316-abaf22ebdef9\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " Nov 02 22:13:15.190020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.189905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx\" (UniqueName: \"kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx\") pod \"24c163b3-f8a5-485e-b316-abaf22ebdef9\" (UID: \"24c163b3-f8a5-485e-b316-abaf22ebdef9\") " Nov 02 22:13:15.190152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.190129 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca" pod "44628257-a764-488d-9128-8a86fa97d47c" (UID: "44628257-a764-488d-9128-8a86fa97d47c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:15.190235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.190217 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca" pod "24c163b3-f8a5-485e-b316-abaf22ebdef9" (UID: "24c163b3-f8a5-485e-b316-abaf22ebdef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:15.190576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.190537 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44628257-a764-488d-9128-8a86fa97d47c-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:15.190576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.190564 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:15.191958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.191931 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6" pod "44628257-a764-488d-9128-8a86fa97d47c" (UID: "44628257-a764-488d-9128-8a86fa97d47c"). InnerVolumeSpecName "kube-api-access-cb8k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:15.192088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.192003 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca" pod "24c163b3-f8a5-485e-b316-abaf22ebdef9" (UID: "24c163b3-f8a5-485e-b316-abaf22ebdef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:15.192409 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.192372 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca" pod "44628257-a764-488d-9128-8a86fa97d47c" (UID: "44628257-a764-488d-9128-8a86fa97d47c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:15.192822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.192797 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx" pod "24c163b3-f8a5-485e-b316-abaf22ebdef9" (UID: "24c163b3-f8a5-485e-b316-abaf22ebdef9"). InnerVolumeSpecName "kube-api-access-ph2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:15.291592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.291556 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8k6\" (UniqueName: \"kubernetes.io/projected/44628257-a764-488d-9128-8a86fa97d47c-kube-api-access-cb8k6\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:15.291592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.291594 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c163b3-f8a5-485e-b316-abaf22ebdef9-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:15.291810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.291608 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph2bx\" (UniqueName: \"kubernetes.io/projected/24c163b3-f8a5-485e-b316-abaf22ebdef9-kube-api-access-ph2bx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:15.291810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.291620 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44628257-a764-488d-9128-8a86fa97d47c-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:15.695645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.695613 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2"] Nov 02 22:13:15.695876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.695822 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" containerID="cri-o://9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5" gracePeriod=30 Nov 02 22:13:15.936402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.936371 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" Nov 02 22:13:15.936605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.936408 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst" event={"ID":"44628257-a764-488d-9128-8a86fa97d47c","Type":"ContainerDied","Data":"438d0817f64b6a2e651dd67467eeaeb1dc7ff9ef7d1d5abae42f8237ad54e89a"} Nov 02 22:13:15.936605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.936483 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438d0817f64b6a2e651dd67467eeaeb1dc7ff9ef7d1d5abae42f8237ad54e89a" Nov 02 22:13:15.938212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.938190 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" Nov 02 22:13:15.938212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.938194 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms" event={"ID":"24c163b3-f8a5-485e-b316-abaf22ebdef9","Type":"ContainerDied","Data":"5cadd299f39b5c941c2123b9c8aa87d8af5e3b0c21d393c182bd63763fc89057"} Nov 02 22:13:15.938409 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:15.938233 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cadd299f39b5c941c2123b9c8aa87d8af5e3b0c21d393c182bd63763fc89057" Nov 02 22:13:16.114558 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.114524 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc"] Nov 02 22:13:16.115145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.115131 5091 cpu_manager.go:414] "containerMap: removing container" podUID="24c163b3-f8a5-485e-b316-abaf22ebdef9" containerName="revoke-keys" Nov 02 22:13:16.115193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.115147 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c163b3-f8a5-485e-b316-abaf22ebdef9" containerName="revoke-keys" Nov 02 22:13:16.115193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.115161 5091 cpu_manager.go:414] "containerMap: removing container" podUID="44628257-a764-488d-9128-8a86fa97d47c" containerName="revoke-keys" Nov 02 22:13:16.115193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.115167 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44628257-a764-488d-9128-8a86fa97d47c" containerName="revoke-keys" Nov 02 22:13:16.115298 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.115290 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c163b3-f8a5-485e-b316-abaf22ebdef9" containerName="revoke-keys" Nov 02 22:13:16.115350 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.115300 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="44628257-a764-488d-9128-8a86fa97d47c" containerName="revoke-keys" Nov 02 22:13:16.119024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.119005 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.172019 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.171992 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:16.172182 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.172059 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:13:16.172524 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.172499 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:16.172601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.172545 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:16.177611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.177583 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc"] Nov 02 22:13:16.202461 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.202391 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6vl\" (UniqueName: \"kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.202581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.202505 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.202581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.202535 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.303646 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.303616 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6vl\" (UniqueName: \"kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.303854 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.303729 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.303854 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.303750 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.304474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.304447 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.306184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.306166 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.360325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.360302 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6vl\" (UniqueName: \"kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.433212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.433183 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:16.812419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.812390 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc"] Nov 02 22:13:16.812583 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:13:16.812557 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9726596_af6c_4482_80a8_475b7dc2a4ea.slice/crio-e2dfebf5ed64913956b3bda837a7a3892ea354718c635b0ab1e9c04e95f15bbc WatchSource:0}: Error finding container e2dfebf5ed64913956b3bda837a7a3892ea354718c635b0ab1e9c04e95f15bbc: Status 404 returned error can't find the container with id e2dfebf5ed64913956b3bda837a7a3892ea354718c635b0ab1e9c04e95f15bbc Nov 02 22:13:16.946993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:16.946959 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" event={"ID":"e9726596-af6c-4482-80a8-475b7dc2a4ea","Type":"ContainerStarted","Data":"e2dfebf5ed64913956b3bda837a7a3892ea354718c635b0ab1e9c04e95f15bbc"} Nov 02 22:13:17.956796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:17.956764 5091 generic.go:356] "Generic (PLEG): container finished" podID="e9726596-af6c-4482-80a8-475b7dc2a4ea" containerID="9f336ca8070497774e3fa45c1ebfe38cb7bd0d34796bbfa4dce4116e390e5541" exitCode=0 Nov 02 22:13:17.957161 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:17.956833 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" event={"ID":"e9726596-af6c-4482-80a8-475b7dc2a4ea","Type":"ContainerDied","Data":"9f336ca8070497774e3fa45c1ebfe38cb7bd0d34796bbfa4dce4116e390e5541"} Nov 02 22:13:18.966146 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:18.966118 5091 generic.go:356] "Generic (PLEG): container finished" podID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerID="43e19846384ce5cda96733c6ff5dbde027f6162a9380aa18610c350ab1ea74c5" exitCode=0 Nov 02 22:13:18.966451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:18.966193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" event={"ID":"048fb8f1-e23d-40ae-bcd4-f873adf95ae8","Type":"ContainerDied","Data":"43e19846384ce5cda96733c6ff5dbde027f6162a9380aa18610c350ab1ea74c5"} Nov 02 22:13:18.966451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:18.966253 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" event={"ID":"048fb8f1-e23d-40ae-bcd4-f873adf95ae8","Type":"ContainerDied","Data":"63480dc52784ade3255f087fead2f7840ff654e64d01689d0f651a0c41c797af"} Nov 02 22:13:18.966451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:18.966281 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63480dc52784ade3255f087fead2f7840ff654e64d01689d0f651a0c41c797af" Nov 02 22:13:18.966451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:18.966294 5091 scope.go:122] "RemoveContainer" containerID="b33bb3a1baf2de332c1888ce7c3659f5e968d9a4d615c07a2447b0dcb192f915" Nov 02 22:13:18.978907 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:18.978873 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:13:19.135220 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135190 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135278 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135302 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135328 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135432 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135473 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135571 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:19.135704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135608 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.135704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135639 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo\") pod \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\" (UID: \"048fb8f1-e23d-40ae-bcd4-f873adf95ae8\") " Nov 02 22:13:19.136049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:19.136049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135734 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:19.136049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.135854 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:19.136227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.136171 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:19.136337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.136315 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.136419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.136342 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.136419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.136358 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.136419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.136371 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.136419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.136379 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.137949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.137928 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:13:19.138056 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.137973 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:19.138095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.138078 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "kube-api-access-67jhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:19.138241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.138223 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token" pod "048fb8f1-e23d-40ae-bcd4-f873adf95ae8" (UID: "048fb8f1-e23d-40ae-bcd4-f873adf95ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:19.143854 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.143833 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:19.237726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.237647 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca\") pod \"e9726596-af6c-4482-80a8-475b7dc2a4ea\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " Nov 02 22:13:19.237726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.237717 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl\" (UniqueName: \"kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl\") pod \"e9726596-af6c-4482-80a8-475b7dc2a4ea\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " Nov 02 22:13:19.237964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.237741 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca\") pod \"e9726596-af6c-4482-80a8-475b7dc2a4ea\" (UID: \"e9726596-af6c-4482-80a8-475b7dc2a4ea\") " Nov 02 22:13:19.238076 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.238053 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca" pod "e9726596-af6c-4482-80a8-475b7dc2a4ea" (UID: "e9726596-af6c-4482-80a8-475b7dc2a4ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:19.238247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.238231 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.238319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.238253 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.238319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.238268 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67jhs\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-kube-api-access-67jhs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.238319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.238283 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.238319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.238297 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048fb8f1-e23d-40ae-bcd4-f873adf95ae8-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.239836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.239809 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca" pod "e9726596-af6c-4482-80a8-475b7dc2a4ea" (UID: "e9726596-af6c-4482-80a8-475b7dc2a4ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:19.239932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.239880 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl" pod "e9726596-af6c-4482-80a8-475b7dc2a4ea" (UID: "e9726596-af6c-4482-80a8-475b7dc2a4ea"). InnerVolumeSpecName "kube-api-access-vw6vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:19.339205 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.339169 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6vl\" (UniqueName: \"kubernetes.io/projected/e9726596-af6c-4482-80a8-475b7dc2a4ea-kube-api-access-vw6vl\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.339205 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.339200 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9726596-af6c-4482-80a8-475b7dc2a4ea-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:19.975430 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.975354 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9" Nov 02 22:13:19.976858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.976832 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" event={"ID":"e9726596-af6c-4482-80a8-475b7dc2a4ea","Type":"ContainerDied","Data":"e2dfebf5ed64913956b3bda837a7a3892ea354718c635b0ab1e9c04e95f15bbc"} Nov 02 22:13:19.976965 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.976860 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc" Nov 02 22:13:19.976965 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:19.976871 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2dfebf5ed64913956b3bda837a7a3892ea354718c635b0ab1e9c04e95f15bbc" Nov 02 22:13:20.303128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.303046 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9"] Nov 02 22:13:20.307624 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.307597 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17435d-openshift-default-5f7dcbcdf5-tqjb9"] Nov 02 22:13:20.322349 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.322323 5091 scope.go:122] "RemoveContainer" containerID="0f2fe8fa2e9f498112a819e122461b4f3581a27e4718e033941e4451c61b47f4" Nov 02 22:13:20.333495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.333466 5091 scope.go:122] "RemoveContainer" containerID="b3540120a820d5e5cbd247e9058e2f5a53a33abdcbbd4de1a103639ee3da3d4d" Nov 02 22:13:20.965696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.965609 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 22:13:20.977226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.977198 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" path="/var/lib/kubelet/pods/048fb8f1-e23d-40ae-bcd4-f873adf95ae8/volumes" Nov 02 22:13:20.985034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.985010 5091 generic.go:356] "Generic (PLEG): container finished" podID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerID="9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5" exitCode=0 Nov 02 22:13:20.985184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.985050 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" Nov 02 22:13:20.985184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.985086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" event={"ID":"1a59afb9-6bdc-4761-89f6-931cee41d27b","Type":"ContainerDied","Data":"9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5"} Nov 02 22:13:20.985184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.985140 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" event={"ID":"1a59afb9-6bdc-4761-89f6-931cee41d27b","Type":"ContainerDied","Data":"2ff8bd7eb4bc9c59ef93fa763699798d420626459d6e217fe4d95d73457d6083"} Nov 02 22:13:20.985184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.985165 5091 scope.go:122] "RemoveContainer" containerID="9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5" Nov 02 22:13:20.995244 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.995228 5091 scope.go:122] "RemoveContainer" containerID="9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5" Nov 02 22:13:20.995527 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:13:20.995510 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5\": container with ID starting with 9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5 not found: ID does not exist" containerID="9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5" Nov 02 22:13:20.995572 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:20.995535 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5"} err="failed to get container status \"9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5\": rpc error: code = NotFound desc = could not find container \"9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5\": container with ID starting with 9e53a6e82c735f58da435b964a3c9b0de0d29a752ae27164066ce354cf3c85e5 not found: ID does not exist" Nov 02 22:13:21.060956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.060923 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.060986 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061048 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061072 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061293 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061120 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061293 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061146 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061293 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061254 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061313 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061387 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data\") pod \"1a59afb9-6bdc-4761-89f6-931cee41d27b\" (UID: \"1a59afb9-6bdc-4761-89f6-931cee41d27b\") " Nov 02 22:13:21.061457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061389 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.061630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061468 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.061630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061539 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.061852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061807 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.062019 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061848 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:21.062019 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061981 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.062019 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.061997 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.062019 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.062007 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.062260 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.062029 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.062260 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.062044 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a59afb9-6bdc-4761-89f6-931cee41d27b-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.063444 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.063417 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.063557 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.063522 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:21.063625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.063613 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:13:21.063822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.063804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw" pod "1a59afb9-6bdc-4761-89f6-931cee41d27b" (UID: "1a59afb9-6bdc-4761-89f6-931cee41d27b"). InnerVolumeSpecName "kube-api-access-mdwjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:21.162999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.162966 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.162999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.162996 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.163191 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.163012 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a59afb9-6bdc-4761-89f6-931cee41d27b-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.163191 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.163025 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwjw\" (UniqueName: \"kubernetes.io/projected/1a59afb9-6bdc-4761-89f6-931cee41d27b-kube-api-access-mdwjw\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.406765 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.406732 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2"] Nov 02 22:13:21.411783 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.411754 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2"] Nov 02 22:13:21.959988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.959954 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dzsk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:21.960164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:21.960023 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dzsk2" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:22.977798 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:22.977764 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" path="/var/lib/kubelet/pods/1a59afb9-6bdc-4761-89f6-931cee41d27b/volumes" Nov 02 22:13:30.723924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.723885 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw"] Nov 02 22:13:30.725484 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.725453 5091 cpu_manager.go:414] "containerMap: removing container" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" Nov 02 22:13:30.725685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.725651 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" Nov 02 22:13:30.725847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.725832 5091 cpu_manager.go:414] "containerMap: removing container" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" Nov 02 22:13:30.725947 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.725931 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" Nov 02 22:13:30.726095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726068 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" Nov 02 22:13:30.726095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726087 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" Nov 02 22:13:30.726277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726108 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e9726596-af6c-4482-80a8-475b7dc2a4ea" containerName="revoke-keys" Nov 02 22:13:30.726277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726117 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9726596-af6c-4482-80a8-475b7dc2a4ea" containerName="revoke-keys" Nov 02 22:13:30.726393 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726285 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a59afb9-6bdc-4761-89f6-931cee41d27b" containerName="istio-proxy" Nov 02 22:13:30.726393 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726360 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9726596-af6c-4482-80a8-475b7dc2a4ea" containerName="revoke-keys" Nov 02 22:13:30.726509 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.726414 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" Nov 02 22:13:30.732407 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.732376 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801211 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801363 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801408 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801479 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801551 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801581 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801617 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801721 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.803692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.801750 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvrc\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.848913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.848873 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw"] Nov 02 22:13:30.903249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903215 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903264 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903378 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903406 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903522 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903555 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvrc\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903628 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903647 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.903918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.903881 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.904111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.904093 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.904279 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.904263 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.904434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.904411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.906067 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.906040 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.906067 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.906063 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.959641 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.959610 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:30.960405 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:30.960386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvrc\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc\") pod \"maas-default-gateway-openshift-default-65c448b547-974qw\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:31.080061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:31.079961 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:31.509686 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:31.509642 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw"] Nov 02 22:13:31.510157 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:13:31.510133 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2698ccdd_d9bc_4566_8665_56839a2e6e86.slice/crio-9f5f0a0a71a6a77a148c4769b24065be3178f0bd61d25e72828636f76c4dc060 WatchSource:0}: Error finding container 9f5f0a0a71a6a77a148c4769b24065be3178f0bd61d25e72828636f76c4dc060: Status 404 returned error can't find the container with id 9f5f0a0a71a6a77a148c4769b24065be3178f0bd61d25e72828636f76c4dc060 Nov 02 22:13:31.512709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:31.512665 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:31.512791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:31.512742 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:31.512791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:31.512769 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:32.216696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:32.204058 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" event={"ID":"2698ccdd-d9bc-4566-8665-56839a2e6e86","Type":"ContainerStarted","Data":"cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded"} Nov 02 22:13:32.217305 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:32.217272 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" event={"ID":"2698ccdd-d9bc-4566-8665-56839a2e6e86","Type":"ContainerStarted","Data":"9f5f0a0a71a6a77a148c4769b24065be3178f0bd61d25e72828636f76c4dc060"} Nov 02 22:13:32.324977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:32.324925 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podStartSLOduration=2.324907726 podStartE2EDuration="2.324907726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:32.323065961 +0000 UTC m=+2359.485161081" watchObservedRunningTime="2026-11-02 22:13:32.324907726 +0000 UTC m=+2359.487002845" Nov 02 22:13:33.083969 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:33.083927 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:33.084395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:33.084369 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:33.084559 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:33.084535 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:34.088774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:34.084337 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:34.088774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:34.084394 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:34.193814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:34.193762 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75"] Nov 02 22:13:34.197137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:34.197115 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc145-mvd75"] Nov 02 22:13:35.008898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:35.008855 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b12d95-3e27-40d0-a989-dd11fb78de5f" path="/var/lib/kubelet/pods/92b12d95-3e27-40d0-a989-dd11fb78de5f/volumes" Nov 02 22:13:35.084083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:35.084044 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:35.084333 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:35.084304 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:36.087694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:36.084030 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:36.087694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:36.084109 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:37.086694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:37.084011 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:37.086694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:37.084151 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:37.116445 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:13:37.116412 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:13:38.080972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:38.080937 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:38.081227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:38.080994 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:39.080994 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:39.080963 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-974qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:13:39.081375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:39.081012 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 02 22:13:40.222002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:40.221962 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:40.223193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:40.223164 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:40.223784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:40.223764 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:13:40.921412 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:40.921377 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5"] Nov 02 22:13:40.921698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:40.921640 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" containerID="cri-o://88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f" gracePeriod=30 Nov 02 22:13:46.193661 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.193605 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:13:46.264961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.264935 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.264980 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265017 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265046 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265150 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265186 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265281 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265341 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265470 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265370 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token\") pod \"1bfc1424-b3de-491f-a94f-c9408f14b660\" (UID: \"1bfc1424-b3de-491f-a94f-c9408f14b660\") " Nov 02 22:13:46.265764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265350 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:46.265764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265438 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:46.265764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265510 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:46.265764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265547 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:46.265764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.265715 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:46.266094 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.266069 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.266164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.266101 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfc1424-b3de-491f-a94f-c9408f14b660-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.266164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.266118 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.266164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.266132 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.266164 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.266148 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.267368 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.267346 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:13:46.267457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.267435 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "kube-api-access-5tph5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:46.267849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.267828 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:46.267906 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.267858 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token" pod "1bfc1424-b3de-491f-a94f-c9408f14b660" (UID: "1bfc1424-b3de-491f-a94f-c9408f14b660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:46.368187 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.368156 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.368187 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.368188 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.368366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.368203 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfc1424-b3de-491f-a94f-c9408f14b660-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.368366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.368218 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tph5\" (UniqueName: \"kubernetes.io/projected/1bfc1424-b3de-491f-a94f-c9408f14b660-kube-api-access-5tph5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:46.374775 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.374740 5091 generic.go:356] "Generic (PLEG): container finished" podID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerID="88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f" exitCode=0 Nov 02 22:13:46.374944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.374794 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" Nov 02 22:13:46.374944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.374827 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" event={"ID":"1bfc1424-b3de-491f-a94f-c9408f14b660","Type":"ContainerDied","Data":"88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f"} Nov 02 22:13:46.374944 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.374902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5" event={"ID":"1bfc1424-b3de-491f-a94f-c9408f14b660","Type":"ContainerDied","Data":"c10c9abd05fa3cb30f16085bc24dfc51b1fa088ecdb31fe5842e91b9cdd5176f"} Nov 02 22:13:46.375122 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.374950 5091 scope.go:122] "RemoveContainer" containerID="88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f" Nov 02 22:13:46.393600 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.393578 5091 scope.go:122] "RemoveContainer" containerID="2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61" Nov 02 22:13:46.403809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.403793 5091 scope.go:122] "RemoveContainer" containerID="88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f" Nov 02 22:13:46.404154 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:13:46.404129 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f\": container with ID starting with 88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f not found: ID does not exist" containerID="88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f" Nov 02 22:13:46.404249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.404166 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f"} err="failed to get container status \"88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f\": rpc error: code = NotFound desc = could not find container \"88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f\": container with ID starting with 88fa23762349483292b70fb5c76e465b9a1793e18150d3426c5f26e8cae5424f not found: ID does not exist" Nov 02 22:13:46.404249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.404189 5091 scope.go:122] "RemoveContainer" containerID="2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61" Nov 02 22:13:46.404493 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:13:46.404465 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61\": container with ID starting with 2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61 not found: ID does not exist" containerID="2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61" Nov 02 22:13:46.404539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.404497 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61"} err="failed to get container status \"2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61\": rpc error: code = NotFound desc = could not find container \"2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61\": container with ID starting with 2672f532a273a94c2dab3813db502dfbe8618754bc99a09205259f600a6a0d61 not found: ID does not exist" Nov 02 22:13:46.519701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.519607 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5"] Nov 02 22:13:46.525290 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.525265 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdc145-openshift-default-78fbf5b4dc-srzq5"] Nov 02 22:13:46.977513 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:46.977474 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" path="/var/lib/kubelet/pods/1bfc1424-b3de-491f-a94f-c9408f14b660/volumes" Nov 02 22:13:56.368414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:56.368334 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz"] Nov 02 22:13:56.417784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:56.417748 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-027bdf-nf9nz"] Nov 02 22:13:56.977948 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:13:56.977904 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3aa39f-91e1-4af6-af2a-0b5c3286de4f" path="/var/lib/kubelet/pods/ac3aa39f-91e1-4af6-af2a-0b5c3286de4f/volumes" Nov 02 22:14:02.756420 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:02.756384 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj"] Nov 02 22:14:02.805654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:02.805621 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2a9be7-qj2bj"] Nov 02 22:14:02.977636 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:02.977603 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49b1b39-a659-4a94-9305-bdd50ce239fb" path="/var/lib/kubelet/pods/e49b1b39-a659-4a94-9305-bdd50ce239fb/volumes" Nov 02 22:14:05.064065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064033 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv"] Nov 02 22:14:05.064621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064607 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" Nov 02 22:14:05.064685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064622 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" Nov 02 22:14:05.064685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064638 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" Nov 02 22:14:05.064685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064643 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" Nov 02 22:14:05.064785 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064761 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="048fb8f1-e23d-40ae-bcd4-f873adf95ae8" containerName="istio-proxy" Nov 02 22:14:05.064785 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.064772 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" Nov 02 22:14:05.069880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.069864 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.125145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.125117 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv"] Nov 02 22:14:05.161351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.161317 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.161516 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.161377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5l8\" (UniqueName: \"kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.161623 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.161598 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.169826 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.169800 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:05.169953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.169932 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:05.169953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.169947 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:14:05.171289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.171272 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:05.262628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.262592 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.262860 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.262657 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5l8\" (UniqueName: \"kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.262860 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.262835 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.263521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.263465 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.265189 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.265169 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.318573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.318518 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5l8\" (UniqueName: \"kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.383596 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.383574 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:05.757224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:05.757186 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv"] Nov 02 22:14:05.759037 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:05.759003 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469d3c45_f298_4d7c_8ecb_16cf0d29b836.slice/crio-71ec019c8e9eab52c7adbde86c3ab55e6737f93af91f870a5d26abbbfa9bb24d WatchSource:0}: Error finding container 71ec019c8e9eab52c7adbde86c3ab55e6737f93af91f870a5d26abbbfa9bb24d: Status 404 returned error can't find the container with id 71ec019c8e9eab52c7adbde86c3ab55e6737f93af91f870a5d26abbbfa9bb24d Nov 02 22:14:06.542881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:06.542786 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" event={"ID":"469d3c45-f298-4d7c-8ecb-16cf0d29b836","Type":"ContainerStarted","Data":"c9e490aac68e29a0d68c77c6ed1993f825825a801e7fc59c04b5bfac5af2b502"} Nov 02 22:14:06.542881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:06.542842 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" event={"ID":"469d3c45-f298-4d7c-8ecb-16cf0d29b836","Type":"ContainerStarted","Data":"71ec019c8e9eab52c7adbde86c3ab55e6737f93af91f870a5d26abbbfa9bb24d"} Nov 02 22:14:06.656622 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:06.656569 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" podStartSLOduration=1.656552711 podStartE2EDuration="1.656552711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:06.654275843 +0000 UTC m=+2393.816370963" watchObservedRunningTime="2026-11-02 22:14:06.656552711 +0000 UTC m=+2393.818647830" Nov 02 22:14:07.113749 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:07.113715 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:14:07.551219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:07.551180 5091 generic.go:356] "Generic (PLEG): container finished" podID="469d3c45-f298-4d7c-8ecb-16cf0d29b836" containerID="c9e490aac68e29a0d68c77c6ed1993f825825a801e7fc59c04b5bfac5af2b502" exitCode=0 Nov 02 22:14:07.551686 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:07.551264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" event={"ID":"469d3c45-f298-4d7c-8ecb-16cf0d29b836","Type":"ContainerDied","Data":"c9e490aac68e29a0d68c77c6ed1993f825825a801e7fc59c04b5bfac5af2b502"} Nov 02 22:14:08.713007 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.712980 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:08.804742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.804714 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca\") pod \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " Nov 02 22:14:08.804887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.804785 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8\" (UniqueName: \"kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8\") pod \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " Nov 02 22:14:08.804887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.804861 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca\") pod \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\" (UID: \"469d3c45-f298-4d7c-8ecb-16cf0d29b836\") " Nov 02 22:14:08.805175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.805141 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca" pod "469d3c45-f298-4d7c-8ecb-16cf0d29b836" (UID: "469d3c45-f298-4d7c-8ecb-16cf0d29b836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:08.805562 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.805507 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:08.807089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.807059 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca" pod "469d3c45-f298-4d7c-8ecb-16cf0d29b836" (UID: "469d3c45-f298-4d7c-8ecb-16cf0d29b836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:08.807169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.807127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8" pod "469d3c45-f298-4d7c-8ecb-16cf0d29b836" (UID: "469d3c45-f298-4d7c-8ecb-16cf0d29b836"). InnerVolumeSpecName "kube-api-access-6n5l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:08.907241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.907207 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/469d3c45-f298-4d7c-8ecb-16cf0d29b836-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:08.907241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:08.907235 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5l8\" (UniqueName: \"kubernetes.io/projected/469d3c45-f298-4d7c-8ecb-16cf0d29b836-kube-api-access-6n5l8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:09.536792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:09.536751 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp"] Nov 02 22:14:09.553031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:09.552994 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb"] Nov 02 22:14:09.577319 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:09.577279 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" event={"ID":"469d3c45-f298-4d7c-8ecb-16cf0d29b836","Type":"ContainerDied","Data":"71ec019c8e9eab52c7adbde86c3ab55e6737f93af91f870a5d26abbbfa9bb24d"} Nov 02 22:14:09.577594 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:09.577566 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ec019c8e9eab52c7adbde86c3ab55e6737f93af91f870a5d26abbbfa9bb24d" Nov 02 22:14:09.577594 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:09.577335 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv" Nov 02 22:14:10.359563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.359533 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp"] Nov 02 22:14:10.360139 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.360124 5091 cpu_manager.go:414] "containerMap: removing container" podUID="469d3c45-f298-4d7c-8ecb-16cf0d29b836" containerName="revoke-keys" Nov 02 22:14:10.360184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.360140 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469d3c45-f298-4d7c-8ecb-16cf0d29b836" containerName="revoke-keys" Nov 02 22:14:10.360273 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.360260 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="469d3c45-f298-4d7c-8ecb-16cf0d29b836" containerName="revoke-keys" Nov 02 22:14:10.360312 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.360278 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfc1424-b3de-491f-a94f-c9408f14b660" containerName="istio-proxy" Nov 02 22:14:10.363900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.363880 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.420721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.420693 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp"] Nov 02 22:14:10.465413 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.465386 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:10.465572 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.465438 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:14:10.465572 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.465465 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:10.465572 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.465470 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:10.528240 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.528208 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.528420 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.528252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.528495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.528448 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4k5\" (UniqueName: \"kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.629325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.629294 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4k5\" (UniqueName: \"kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.629475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.629435 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.629528 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.629482 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.630115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.630087 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.631885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.631863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.685258 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.685232 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4k5\" (UniqueName: \"kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5\") pod \"maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:10.977525 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:10.977439 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:11.359749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.359715 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp"] Nov 02 22:14:11.360637 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:11.360611 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc976f4bf_44ce_432d_b554_8db925933215.slice/crio-a5b7aa62855edd7c66ec6e1622d71895fdf5a8caef8fde5d5bc0efeb1311a42b WatchSource:0}: Error finding container a5b7aa62855edd7c66ec6e1622d71895fdf5a8caef8fde5d5bc0efeb1311a42b: Status 404 returned error can't find the container with id a5b7aa62855edd7c66ec6e1622d71895fdf5a8caef8fde5d5bc0efeb1311a42b Nov 02 22:14:11.592208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.592177 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" event={"ID":"c976f4bf-44ce-432d-b554-8db925933215","Type":"ContainerStarted","Data":"a5b7aa62855edd7c66ec6e1622d71895fdf5a8caef8fde5d5bc0efeb1311a42b"} Nov 02 22:14:11.780045 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.780007 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm"] Nov 02 22:14:11.786987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.786954 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:11.840849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.840808 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm"] Nov 02 22:14:11.948066 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.947953 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htp2\" (UniqueName: \"kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:11.948227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.948069 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:11.948227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:11.948117 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.049069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.049026 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.049265 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.049248 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htp2\" (UniqueName: \"kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.049334 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.049288 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.049975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.049952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.051393 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.051373 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.106190 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.106164 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htp2\" (UniqueName: \"kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2\") pod \"maas-api-revoke-sub-180a8a31-n7knm\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.402680 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.402637 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:12.601063 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.600977 5091 generic.go:356] "Generic (PLEG): container finished" podID="c976f4bf-44ce-432d-b554-8db925933215" containerID="75c92579e71f1a35b5fd423c2c90c461cbd48b9538ae253e99f37e98e80a54a5" exitCode=0 Nov 02 22:14:12.601213 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.601056 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" event={"ID":"c976f4bf-44ce-432d-b554-8db925933215","Type":"ContainerDied","Data":"75c92579e71f1a35b5fd423c2c90c461cbd48b9538ae253e99f37e98e80a54a5"} Nov 02 22:14:12.786886 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:12.786855 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm"] Nov 02 22:14:12.796097 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:12.795974 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352f4baf_0508_4307_9d95_c903c98dc9ca.slice/crio-cda9007fcb5bf1655aa8763260c2d468567f41045fd82a303f47bc05f7d86323 WatchSource:0}: Error finding container cda9007fcb5bf1655aa8763260c2d468567f41045fd82a303f47bc05f7d86323: Status 404 returned error can't find the container with id cda9007fcb5bf1655aa8763260c2d468567f41045fd82a303f47bc05f7d86323 Nov 02 22:14:13.448760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.448735 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:14:13.452414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.452390 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:14:13.609858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.609820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" event={"ID":"352f4baf-0508-4307-9d95-c903c98dc9ca","Type":"ContainerStarted","Data":"2c7e6f7b6c14fa8a0855d5fefd4e6af42f4f95ef15888acfc53d6f16cc51aa13"} Nov 02 22:14:13.610046 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.609876 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" event={"ID":"352f4baf-0508-4307-9d95-c903c98dc9ca","Type":"ContainerStarted","Data":"cda9007fcb5bf1655aa8763260c2d468567f41045fd82a303f47bc05f7d86323"} Nov 02 22:14:13.723023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.722930 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" podStartSLOduration=2.7229129309999998 podStartE2EDuration="2.722912931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:13.721784583 +0000 UTC m=+2400.883879702" watchObservedRunningTime="2026-11-02 22:14:13.722912931 +0000 UTC m=+2400.885008051" Nov 02 22:14:13.760958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.760934 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:13.871703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.871643 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca\") pod \"c976f4bf-44ce-432d-b554-8db925933215\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " Nov 02 22:14:13.871868 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.871849 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5\" (UniqueName: \"kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5\") pod \"c976f4bf-44ce-432d-b554-8db925933215\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " Nov 02 22:14:13.871936 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.871900 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca\") pod \"c976f4bf-44ce-432d-b554-8db925933215\" (UID: \"c976f4bf-44ce-432d-b554-8db925933215\") " Nov 02 22:14:13.872276 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.872254 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca" pod "c976f4bf-44ce-432d-b554-8db925933215" (UID: "c976f4bf-44ce-432d-b554-8db925933215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:13.872403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.872389 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c976f4bf-44ce-432d-b554-8db925933215-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:13.874170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.874123 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5" pod "c976f4bf-44ce-432d-b554-8db925933215" (UID: "c976f4bf-44ce-432d-b554-8db925933215"). InnerVolumeSpecName "kube-api-access-fx4k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:13.874433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.874208 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca" pod "c976f4bf-44ce-432d-b554-8db925933215" (UID: "c976f4bf-44ce-432d-b554-8db925933215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:13.973734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.973622 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c976f4bf-44ce-432d-b554-8db925933215-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:13.973734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:13.973654 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4k5\" (UniqueName: \"kubernetes.io/projected/c976f4bf-44ce-432d-b554-8db925933215-kube-api-access-fx4k5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.617900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.617870 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" Nov 02 22:14:14.618405 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.617902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp" event={"ID":"c976f4bf-44ce-432d-b554-8db925933215","Type":"ContainerDied","Data":"a5b7aa62855edd7c66ec6e1622d71895fdf5a8caef8fde5d5bc0efeb1311a42b"} Nov 02 22:14:14.618405 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.617994 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b7aa62855edd7c66ec6e1622d71895fdf5a8caef8fde5d5bc0efeb1311a42b" Nov 02 22:14:14.619565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.619539 5091 generic.go:356] "Generic (PLEG): container finished" podID="352f4baf-0508-4307-9d95-c903c98dc9ca" containerID="2c7e6f7b6c14fa8a0855d5fefd4e6af42f4f95ef15888acfc53d6f16cc51aa13" exitCode=0 Nov 02 22:14:14.619728 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.619571 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" event={"ID":"352f4baf-0508-4307-9d95-c903c98dc9ca","Type":"ContainerDied","Data":"2c7e6f7b6c14fa8a0855d5fefd4e6af42f4f95ef15888acfc53d6f16cc51aa13"} Nov 02 22:14:14.786601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.786570 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw"] Nov 02 22:14:14.811368 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:14.811335 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9"] Nov 02 22:14:15.793532 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.793510 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:15.899002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.898940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca\") pod \"352f4baf-0508-4307-9d95-c903c98dc9ca\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " Nov 02 22:14:15.899140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.899031 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2\" (UniqueName: \"kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2\") pod \"352f4baf-0508-4307-9d95-c903c98dc9ca\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " Nov 02 22:14:15.899140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.899092 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca\") pod \"352f4baf-0508-4307-9d95-c903c98dc9ca\" (UID: \"352f4baf-0508-4307-9d95-c903c98dc9ca\") " Nov 02 22:14:15.899428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.899404 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca" pod "352f4baf-0508-4307-9d95-c903c98dc9ca" (UID: "352f4baf-0508-4307-9d95-c903c98dc9ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:15.899543 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.899526 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:15.901176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.901153 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2" pod "352f4baf-0508-4307-9d95-c903c98dc9ca" (UID: "352f4baf-0508-4307-9d95-c903c98dc9ca"). InnerVolumeSpecName "kube-api-access-2htp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:15.901274 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:15.901196 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca" pod "352f4baf-0508-4307-9d95-c903c98dc9ca" (UID: "352f4baf-0508-4307-9d95-c903c98dc9ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:16.000068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:16.000040 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htp2\" (UniqueName: \"kubernetes.io/projected/352f4baf-0508-4307-9d95-c903c98dc9ca-kube-api-access-2htp2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:16.000068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:16.000065 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352f4baf-0508-4307-9d95-c903c98dc9ca-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:16.637066 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:16.637035 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" Nov 02 22:14:16.637217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:16.637064 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm" event={"ID":"352f4baf-0508-4307-9d95-c903c98dc9ca","Type":"ContainerDied","Data":"cda9007fcb5bf1655aa8763260c2d468567f41045fd82a303f47bc05f7d86323"} Nov 02 22:14:16.637217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:16.637105 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda9007fcb5bf1655aa8763260c2d468567f41045fd82a303f47bc05f7d86323" Nov 02 22:14:16.675322 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:16.675290 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh"] Nov 02 22:14:19.510032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:19.509996 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x"] Nov 02 22:14:19.558774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:19.558743 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f48c4d-s8r8x"] Nov 02 22:14:20.389682 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:20.389639 5091 scope.go:122] "RemoveContainer" containerID="48c70bc37be1c3135e3eb7e124c1166f5c4d631f71cb5a7e53020c104988ff50" Nov 02 22:14:20.400136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:20.400117 5091 scope.go:122] "RemoveContainer" containerID="0cdfb0defe8627f5f413de6a777adbf3aa1785df2a9a0c886a1f42bf2e3478eb" Nov 02 22:14:20.978152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:20.978114 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8137518d-9f8a-4f9f-90e1-565422e58f87" path="/var/lib/kubelet/pods/8137518d-9f8a-4f9f-90e1-565422e58f87/volumes" Nov 02 22:14:21.675997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:21.675960 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerName="payload-pre-processing" containerID="cri-o://f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff" gracePeriod=25 Nov 02 22:14:21.817361 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:21.817329 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f"] Nov 02 22:14:22.966429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.966366 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb"] Nov 02 22:14:22.967230 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.967207 5091 cpu_manager.go:414] "containerMap: removing container" podUID="352f4baf-0508-4307-9d95-c903c98dc9ca" containerName="revoke-keys" Nov 02 22:14:22.967339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.967230 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352f4baf-0508-4307-9d95-c903c98dc9ca" containerName="revoke-keys" Nov 02 22:14:22.967339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.967252 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c976f4bf-44ce-432d-b554-8db925933215" containerName="revoke-keys" Nov 02 22:14:22.967339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.967260 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c976f4bf-44ce-432d-b554-8db925933215" containerName="revoke-keys" Nov 02 22:14:22.967502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.967435 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="352f4baf-0508-4307-9d95-c903c98dc9ca" containerName="revoke-keys" Nov 02 22:14:22.967502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.967451 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c976f4bf-44ce-432d-b554-8db925933215" containerName="revoke-keys" Nov 02 22:14:22.973440 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:22.973421 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.007977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.007948 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhplq\" (UniqueName: \"kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.008092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.008071 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.008151 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.008127 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.025751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.025713 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:23.025943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.025731 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:23.025943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.025901 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:23.026539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.026518 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:14:23.026752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.026729 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb"] Nov 02 22:14:23.109051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.109017 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.109208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.109156 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhplq\" (UniqueName: \"kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.109264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.109209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.109843 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.109820 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.111377 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.111357 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.165047 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.165022 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhplq\" (UniqueName: \"kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq\") pod \"maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.295745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.289196 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:23.735702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.735408 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb"] Nov 02 22:14:23.737846 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:23.737816 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb19f9ba_a147_4d83_9edf_99c569468d3e.slice/crio-87acde929639a7f7b05a483b224098e471ea0e44c9a63fada668ebee08ed6816 WatchSource:0}: Error finding container 87acde929639a7f7b05a483b224098e471ea0e44c9a63fada668ebee08ed6816: Status 404 returned error can't find the container with id 87acde929639a7f7b05a483b224098e471ea0e44c9a63fada668ebee08ed6816 Nov 02 22:14:23.747386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.747363 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm"] Nov 02 22:14:23.757032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.757014 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.801312 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.801283 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d"] Nov 02 22:14:23.805272 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.805255 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:23.826607 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.826585 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgpld\" (UniqueName: \"kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.826726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.826656 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.826783 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.826766 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.846879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.846855 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm"] Nov 02 22:14:23.866431 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.866404 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d"] Nov 02 22:14:23.942414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.941480 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:23.942414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.941556 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2fj\" (UniqueName: \"kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:23.942414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.941612 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgpld\" (UniqueName: \"kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.942414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.941644 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:23.942414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.941789 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.942414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.941865 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.944202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.944177 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.946730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.946076 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:23.999243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:23.999172 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgpld\" (UniqueName: \"kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld\") pod \"maas-api-revoke-sub-29f95790-rpwjm\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:24.043373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.043336 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.043538 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.043388 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2fj\" (UniqueName: \"kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.043538 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.043447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.044119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.044093 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.046247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.046228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.087615 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.087591 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:24.102898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.102875 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2fj\" (UniqueName: \"kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj\") pod \"maas-api-revoke-sub-2a91e24c-kjr7d\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.118529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.118498 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:24.491749 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:24.491713 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66263ea9_bb12_4968_83a6_ffc87473abb8.slice/crio-e42db82e6f4a6f7730bea36f573db8be236694873eb948b684d95f7060500264 WatchSource:0}: Error finding container e42db82e6f4a6f7730bea36f573db8be236694873eb948b684d95f7060500264: Status 404 returned error can't find the container with id e42db82e6f4a6f7730bea36f573db8be236694873eb948b684d95f7060500264 Nov 02 22:14:24.493050 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.493022 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm"] Nov 02 22:14:24.585005 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.584959 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" containerID="cri-o://a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715" gracePeriod=45 Nov 02 22:14:24.587388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.587331 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" containerID="cri-o://dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1" gracePeriod=45 Nov 02 22:14:24.658678 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:24.658627 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7896c66_d594_421a_8c95_91559ccbea35.slice/crio-conmon-a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:14:24.680389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.680366 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d"] Nov 02 22:14:24.708999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.708801 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" event={"ID":"66263ea9-bb12-4968-83a6-ffc87473abb8","Type":"ContainerStarted","Data":"e42db82e6f4a6f7730bea36f573db8be236694873eb948b684d95f7060500264"} Nov 02 22:14:24.710190 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:24.710166 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e124af_a6ec_4c9f_8a4f_7cb419ad7340.slice/crio-2ca8461cf0392530541056f98adeccb83e6169a767f07c23fbcf363053a48726 WatchSource:0}: Error finding container 2ca8461cf0392530541056f98adeccb83e6169a767f07c23fbcf363053a48726: Status 404 returned error can't find the container with id 2ca8461cf0392530541056f98adeccb83e6169a767f07c23fbcf363053a48726 Nov 02 22:14:24.711458 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.711177 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" event={"ID":"eb19f9ba-a147-4d83-9edf-99c569468d3e","Type":"ContainerStarted","Data":"9e0be7c1beff3a4118ccc9d023b36f2726235875715a127b909461ce173670be"} Nov 02 22:14:24.711458 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.711233 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" event={"ID":"eb19f9ba-a147-4d83-9edf-99c569468d3e","Type":"ContainerStarted","Data":"87acde929639a7f7b05a483b224098e471ea0e44c9a63fada668ebee08ed6816"} Nov 02 22:14:24.735433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.735371 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Nov 02 22:14:24.902709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.893626 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" podStartSLOduration=2.89360584 podStartE2EDuration="2.89360584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:24.860197838 +0000 UTC m=+2412.022292984" watchObservedRunningTime="2026-11-02 22:14:24.89360584 +0000 UTC m=+2412.055700960" Nov 02 22:14:24.941846 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.938832 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 22:14:24.978726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.978699 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j\" (UniqueName: \"kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j\") pod \"146b8ca0-a44d-4f48-a022-a4d335e92135\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " Nov 02 22:14:24.978807 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.978780 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs\" (UniqueName: \"kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs\") pod \"146b8ca0-a44d-4f48-a022-a4d335e92135\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " Nov 02 22:14:24.978870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.978811 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir\") pod \"146b8ca0-a44d-4f48-a022-a4d335e92135\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " Nov 02 22:14:24.978925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.978876 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home\") pod \"146b8ca0-a44d-4f48-a022-a4d335e92135\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " Nov 02 22:14:24.979145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.979040 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm\") pod \"146b8ca0-a44d-4f48-a022-a4d335e92135\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " Nov 02 22:14:24.979145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.979078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache\") pod \"146b8ca0-a44d-4f48-a022-a4d335e92135\" (UID: \"146b8ca0-a44d-4f48-a022-a4d335e92135\") " Nov 02 22:14:24.979653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.979617 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home" pod "146b8ca0-a44d-4f48-a022-a4d335e92135" (UID: "146b8ca0-a44d-4f48-a022-a4d335e92135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:24.979925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.979871 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache" pod "146b8ca0-a44d-4f48-a022-a4d335e92135" (UID: "146b8ca0-a44d-4f48-a022-a4d335e92135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:24.980003 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.979906 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir" pod "146b8ca0-a44d-4f48-a022-a4d335e92135" (UID: "146b8ca0-a44d-4f48-a022-a4d335e92135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:24.982128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.982102 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm" pod "146b8ca0-a44d-4f48-a022-a4d335e92135" (UID: "146b8ca0-a44d-4f48-a022-a4d335e92135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:24.982227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.982182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j" pod "146b8ca0-a44d-4f48-a022-a4d335e92135" (UID: "146b8ca0-a44d-4f48-a022-a4d335e92135"). InnerVolumeSpecName "kube-api-access-mbb2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:24.983502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.983481 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs" pod "146b8ca0-a44d-4f48-a022-a4d335e92135" (UID: "146b8ca0-a44d-4f48-a022-a4d335e92135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:24.990181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:24.990158 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 22:14:25.081036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081004 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt\" (UniqueName: \"kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt\") pod \"f7896c66-d594-421a-8c95-91559ccbea35\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " Nov 02 22:14:25.081760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs\") pod \"f7896c66-d594-421a-8c95-91559ccbea35\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " Nov 02 22:14:25.081760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081116 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home\") pod \"f7896c66-d594-421a-8c95-91559ccbea35\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " Nov 02 22:14:25.081760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081159 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm\") pod \"f7896c66-d594-421a-8c95-91559ccbea35\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " Nov 02 22:14:25.081760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081231 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache\") pod \"f7896c66-d594-421a-8c95-91559ccbea35\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " Nov 02 22:14:25.081760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081440 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir\") pod \"f7896c66-d594-421a-8c95-91559ccbea35\" (UID: \"f7896c66-d594-421a-8c95-91559ccbea35\") " Nov 02 22:14:25.081760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081569 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home" pod "f7896c66-d594-421a-8c95-91559ccbea35" (UID: "f7896c66-d594-421a-8c95-91559ccbea35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.082134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.081991 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache" pod "f7896c66-d594-421a-8c95-91559ccbea35" (UID: "f7896c66-d594-421a-8c95-91559ccbea35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.082134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082037 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir" pod "f7896c66-d594-421a-8c95-91559ccbea35" (UID: "f7896c66-d594-421a-8c95-91559ccbea35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.082134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082127 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146b8ca0-a44d-4f48-a022-a4d335e92135-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082142 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082151 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082159 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082169 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146b8ca0-a44d-4f48-a022-a4d335e92135-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082178 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082192 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.082307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.082206 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbb2j\" (UniqueName: \"kubernetes.io/projected/146b8ca0-a44d-4f48-a022-a4d335e92135-kube-api-access-mbb2j\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.084223 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.084200 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm" pod "f7896c66-d594-421a-8c95-91559ccbea35" (UID: "f7896c66-d594-421a-8c95-91559ccbea35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.086100 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.086077 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt" pod "f7896c66-d594-421a-8c95-91559ccbea35" (UID: "f7896c66-d594-421a-8c95-91559ccbea35"). InnerVolumeSpecName "kube-api-access-rtnpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:25.087217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.087189 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs" pod "f7896c66-d594-421a-8c95-91559ccbea35" (UID: "f7896c66-d594-421a-8c95-91559ccbea35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:25.183390 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.183358 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.183505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.183398 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtnpt\" (UniqueName: \"kubernetes.io/projected/f7896c66-d594-421a-8c95-91559ccbea35-kube-api-access-rtnpt\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.183505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.183429 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7896c66-d594-421a-8c95-91559ccbea35-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.183505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.183444 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7896c66-d594-421a-8c95-91559ccbea35-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.719793 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.719759 5091 generic.go:356] "Generic (PLEG): container finished" podID="66263ea9-bb12-4968-83a6-ffc87473abb8" containerID="5f3a997808f1e011908d784edd008421c48fad74ac4516c50b5a4c02179bb805" exitCode=0 Nov 02 22:14:25.719977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.719844 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" event={"ID":"66263ea9-bb12-4968-83a6-ffc87473abb8","Type":"ContainerDied","Data":"5f3a997808f1e011908d784edd008421c48fad74ac4516c50b5a4c02179bb805"} Nov 02 22:14:25.721423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.721400 5091 generic.go:356] "Generic (PLEG): container finished" podID="eb19f9ba-a147-4d83-9edf-99c569468d3e" containerID="9e0be7c1beff3a4118ccc9d023b36f2726235875715a127b909461ce173670be" exitCode=0 Nov 02 22:14:25.721544 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.721469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" event={"ID":"eb19f9ba-a147-4d83-9edf-99c569468d3e","Type":"ContainerDied","Data":"9e0be7c1beff3a4118ccc9d023b36f2726235875715a127b909461ce173670be"} Nov 02 22:14:25.723016 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.722996 5091 generic.go:356] "Generic (PLEG): container finished" podID="01e124af-a6ec-4c9f-8a4f-7cb419ad7340" containerID="7b40bbc0312f0cde51259457dfef3a80ca781a92c7a9ac3c2df588ee13f2c210" exitCode=0 Nov 02 22:14:25.723113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.723076 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" event={"ID":"01e124af-a6ec-4c9f-8a4f-7cb419ad7340","Type":"ContainerDied","Data":"7b40bbc0312f0cde51259457dfef3a80ca781a92c7a9ac3c2df588ee13f2c210"} Nov 02 22:14:25.723169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.723117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" event={"ID":"01e124af-a6ec-4c9f-8a4f-7cb419ad7340","Type":"ContainerStarted","Data":"2ca8461cf0392530541056f98adeccb83e6169a767f07c23fbcf363053a48726"} Nov 02 22:14:25.724263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.724244 5091 generic.go:356] "Generic (PLEG): container finished" podID="f7896c66-d594-421a-8c95-91559ccbea35" containerID="a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715" exitCode=0 Nov 02 22:14:25.724340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.724286 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" Nov 02 22:14:25.724340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.724316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" event={"ID":"f7896c66-d594-421a-8c95-91559ccbea35","Type":"ContainerDied","Data":"a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715"} Nov 02 22:14:25.724425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.724347 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp" event={"ID":"f7896c66-d594-421a-8c95-91559ccbea35","Type":"ContainerDied","Data":"0136ca39979bb242ab23af2e3ec17bfb8e2d7186d74d2ada41ee869be9f3b5e3"} Nov 02 22:14:25.724425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.724365 5091 scope.go:122] "RemoveContainer" containerID="a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715" Nov 02 22:14:25.725857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.725838 5091 generic.go:356] "Generic (PLEG): container finished" podID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerID="dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1" exitCode=0 Nov 02 22:14:25.725936 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.725891 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" Nov 02 22:14:25.725936 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.725915 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" event={"ID":"146b8ca0-a44d-4f48-a022-a4d335e92135","Type":"ContainerDied","Data":"dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1"} Nov 02 22:14:25.726101 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.725962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb" event={"ID":"146b8ca0-a44d-4f48-a022-a4d335e92135","Type":"ContainerDied","Data":"c67f1c3604b9e2f7f115db86bd7cbacb9c6d6fd66e79b14c58ea902f3231d793"} Nov 02 22:14:25.735352 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.735336 5091 scope.go:122] "RemoveContainer" containerID="a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715" Nov 02 22:14:25.735585 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:25.735560 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715\": container with ID starting with a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715 not found: ID does not exist" containerID="a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715" Nov 02 22:14:25.735717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.735596 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715"} err="failed to get container status \"a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715\": rpc error: code = NotFound desc = could not find container \"a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715\": container with ID starting with a4d4a94c9e22bdaf0117036dc90b7d8f09bc4b3de21986094850b566b55e3715 not found: ID does not exist" Nov 02 22:14:25.735717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.735614 5091 scope.go:122] "RemoveContainer" containerID="dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1" Nov 02 22:14:25.745125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.745105 5091 scope.go:122] "RemoveContainer" containerID="dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1" Nov 02 22:14:25.745412 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:25.745387 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1\": container with ID starting with dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1 not found: ID does not exist" containerID="dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1" Nov 02 22:14:25.745467 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.745420 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1"} err="failed to get container status \"dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1\": rpc error: code = NotFound desc = could not find container \"dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1\": container with ID starting with dde00de7614669f275298e395422e4a09411e56065d0ca962d88eb488c9a07c1 not found: ID does not exist" Nov 02 22:14:25.956887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.956839 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp"] Nov 02 22:14:25.962171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:25.962144 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/distinct-w6-fb8112-kserve-7467697767-7mztp"] Nov 02 22:14:26.052704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.052657 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc"] Nov 02 22:14:26.056194 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.056165 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-caac72-b2sgc"] Nov 02 22:14:26.062996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.062973 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb"] Nov 02 22:14:26.067158 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.067135 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/simulator-w6-fb8112-kserve-665669945-dqsgb"] Nov 02 22:14:26.316771 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.314398 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h"] Nov 02 22:14:26.316771 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.314628 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" containerID="cri-o://2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d" gracePeriod=30 Nov 02 22:14:26.818541 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.818251 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerName="payload-processing" containerID="cri-o://46407fd0f8a8bd4deeed4c62a6d10ea625fa4f8c655464d729545ec65e60b3ba" gracePeriod=25 Nov 02 22:14:26.926133 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.926104 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:26.979226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.979196 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" path="/var/lib/kubelet/pods/146b8ca0-a44d-4f48-a022-a4d335e92135/volumes" Nov 02 22:14:26.979803 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.979783 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9726596-af6c-4482-80a8-475b7dc2a4ea" path="/var/lib/kubelet/pods/e9726596-af6c-4482-80a8-475b7dc2a4ea/volumes" Nov 02 22:14:26.980267 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.980248 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7896c66-d594-421a-8c95-91559ccbea35" path="/var/lib/kubelet/pods/f7896c66-d594-421a-8c95-91559ccbea35/volumes" Nov 02 22:14:26.993709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.993615 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:26.996367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:26.996349 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:27.013709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.013659 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca\") pod \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " Nov 02 22:14:27.013825 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.013806 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj\" (UniqueName: \"kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj\") pod \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " Nov 02 22:14:27.013918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.013905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca\") pod \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\" (UID: \"01e124af-a6ec-4c9f-8a4f-7cb419ad7340\") " Nov 02 22:14:27.014200 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.014178 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca" pod "01e124af-a6ec-4c9f-8a4f-7cb419ad7340" (UID: "01e124af-a6ec-4c9f-8a4f-7cb419ad7340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:27.015713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.015693 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca" pod "01e124af-a6ec-4c9f-8a4f-7cb419ad7340" (UID: "01e124af-a6ec-4c9f-8a4f-7cb419ad7340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:27.015777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.015723 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj" pod "01e124af-a6ec-4c9f-8a4f-7cb419ad7340" (UID: "01e124af-a6ec-4c9f-8a4f-7cb419ad7340"). InnerVolumeSpecName "kube-api-access-5v2fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:27.115410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115376 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca\") pod \"eb19f9ba-a147-4d83-9edf-99c569468d3e\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " Nov 02 22:14:27.115597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115481 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca\") pod \"66263ea9-bb12-4968-83a6-ffc87473abb8\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " Nov 02 22:14:27.115597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115509 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca\") pod \"eb19f9ba-a147-4d83-9edf-99c569468d3e\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " Nov 02 22:14:27.115597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115538 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq\" (UniqueName: \"kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq\") pod \"eb19f9ba-a147-4d83-9edf-99c569468d3e\" (UID: \"eb19f9ba-a147-4d83-9edf-99c569468d3e\") " Nov 02 22:14:27.115597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115582 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca\") pod \"66263ea9-bb12-4968-83a6-ffc87473abb8\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " Nov 02 22:14:27.115887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115648 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld\" (UniqueName: \"kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld\") pod \"66263ea9-bb12-4968-83a6-ffc87473abb8\" (UID: \"66263ea9-bb12-4968-83a6-ffc87473abb8\") " Nov 02 22:14:27.115887 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115746 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca" pod "eb19f9ba-a147-4d83-9edf-99c569468d3e" (UID: "eb19f9ba-a147-4d83-9edf-99c569468d3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:27.116019 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.115998 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca" pod "66263ea9-bb12-4968-83a6-ffc87473abb8" (UID: "66263ea9-bb12-4968-83a6-ffc87473abb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:27.116586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.116510 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.116586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.116541 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.116586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.116557 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.116899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.116595 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.116899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.116617 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2fj\" (UniqueName: \"kubernetes.io/projected/01e124af-a6ec-4c9f-8a4f-7cb419ad7340-kube-api-access-5v2fj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.118067 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.118036 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca" pod "eb19f9ba-a147-4d83-9edf-99c569468d3e" (UID: "eb19f9ba-a147-4d83-9edf-99c569468d3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:27.118271 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.118246 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq" pod "eb19f9ba-a147-4d83-9edf-99c569468d3e" (UID: "eb19f9ba-a147-4d83-9edf-99c569468d3e"). InnerVolumeSpecName "kube-api-access-mhplq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:27.118271 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.118246 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca" pod "66263ea9-bb12-4968-83a6-ffc87473abb8" (UID: "66263ea9-bb12-4968-83a6-ffc87473abb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:27.118962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.118940 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld" pod "66263ea9-bb12-4968-83a6-ffc87473abb8" (UID: "66263ea9-bb12-4968-83a6-ffc87473abb8"). InnerVolumeSpecName "kube-api-access-sgpld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:27.217932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.217897 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66263ea9-bb12-4968-83a6-ffc87473abb8-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.217932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.217925 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb19f9ba-a147-4d83-9edf-99c569468d3e-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.217932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.217936 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhplq\" (UniqueName: \"kubernetes.io/projected/eb19f9ba-a147-4d83-9edf-99c569468d3e-kube-api-access-mhplq\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.218152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.217945 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgpld\" (UniqueName: \"kubernetes.io/projected/66263ea9-bb12-4968-83a6-ffc87473abb8-kube-api-access-sgpld\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:27.463528 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.463491 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:27.464547 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.464513 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs"] Nov 02 22:14:27.473446 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.473421 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd"] Nov 02 22:14:27.748521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.748431 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" Nov 02 22:14:27.748732 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.748430 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm" event={"ID":"66263ea9-bb12-4968-83a6-ffc87473abb8","Type":"ContainerDied","Data":"e42db82e6f4a6f7730bea36f573db8be236694873eb948b684d95f7060500264"} Nov 02 22:14:27.748732 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.748603 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42db82e6f4a6f7730bea36f573db8be236694873eb948b684d95f7060500264" Nov 02 22:14:27.750024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.749999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" event={"ID":"eb19f9ba-a147-4d83-9edf-99c569468d3e","Type":"ContainerDied","Data":"87acde929639a7f7b05a483b224098e471ea0e44c9a63fada668ebee08ed6816"} Nov 02 22:14:27.750192 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.750042 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87acde929639a7f7b05a483b224098e471ea0e44c9a63fada668ebee08ed6816" Nov 02 22:14:27.750192 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.750011 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb" Nov 02 22:14:27.751614 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.751585 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" event={"ID":"01e124af-a6ec-4c9f-8a4f-7cb419ad7340","Type":"ContainerDied","Data":"2ca8461cf0392530541056f98adeccb83e6169a767f07c23fbcf363053a48726"} Nov 02 22:14:27.751730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.751623 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca8461cf0392530541056f98adeccb83e6169a767f07c23fbcf363053a48726" Nov 02 22:14:27.751730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:27.751642 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d" Nov 02 22:14:29.475224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:29.475185 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:29.808035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:29.807939 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" containerID="cri-o://69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007" gracePeriod=45 Nov 02 22:14:29.831421 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:29.831389 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" containerID="cri-o://bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229" gracePeriod=45 Nov 02 22:14:30.105011 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.104981 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 22:14:30.148043 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.148022 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 22:14:30.152247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152227 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb\" (UniqueName: \"kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb\") pod \"92a70dac-c289-41e3-94f2-e522b78fd6c9\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " Nov 02 22:14:30.152324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152309 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home\") pod \"92a70dac-c289-41e3-94f2-e522b78fd6c9\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " Nov 02 22:14:30.152403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152339 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache\") pod \"92a70dac-c289-41e3-94f2-e522b78fd6c9\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " Nov 02 22:14:30.152403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152362 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs\") pod \"92a70dac-c289-41e3-94f2-e522b78fd6c9\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " Nov 02 22:14:30.152403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152378 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir\") pod \"92a70dac-c289-41e3-94f2-e522b78fd6c9\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " Nov 02 22:14:30.152557 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152416 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm\") pod \"92a70dac-c289-41e3-94f2-e522b78fd6c9\" (UID: \"92a70dac-c289-41e3-94f2-e522b78fd6c9\") " Nov 02 22:14:30.152625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home" pod "92a70dac-c289-41e3-94f2-e522b78fd6c9" (UID: "92a70dac-c289-41e3-94f2-e522b78fd6c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.152625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152602 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache" pod "92a70dac-c289-41e3-94f2-e522b78fd6c9" (UID: "92a70dac-c289-41e3-94f2-e522b78fd6c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.152775 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152747 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir" pod "92a70dac-c289-41e3-94f2-e522b78fd6c9" (UID: "92a70dac-c289-41e3-94f2-e522b78fd6c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.153007 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.152986 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.153119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.153007 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.153119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.153022 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.154561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.154531 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm" pod "92a70dac-c289-41e3-94f2-e522b78fd6c9" (UID: "92a70dac-c289-41e3-94f2-e522b78fd6c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.154653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.154592 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs" pod "92a70dac-c289-41e3-94f2-e522b78fd6c9" (UID: "92a70dac-c289-41e3-94f2-e522b78fd6c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:30.154653 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.154618 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb" pod "92a70dac-c289-41e3-94f2-e522b78fd6c9" (UID: "92a70dac-c289-41e3-94f2-e522b78fd6c9"). InnerVolumeSpecName "kube-api-access-8hvcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:30.253549 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.253510 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir\") pod \"b841d1a4-37c8-4403-8448-c61416c6341b\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " Nov 02 22:14:30.253787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.253593 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home\") pod \"b841d1a4-37c8-4403-8448-c61416c6341b\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " Nov 02 22:14:30.253787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.253631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn\" (UniqueName: \"kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn\") pod \"b841d1a4-37c8-4403-8448-c61416c6341b\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " Nov 02 22:14:30.253787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.253712 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache\") pod \"b841d1a4-37c8-4403-8448-c61416c6341b\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " Nov 02 22:14:30.253787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.253750 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm\") pod \"b841d1a4-37c8-4403-8448-c61416c6341b\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " Nov 02 22:14:30.253787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.253786 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs\") pod \"b841d1a4-37c8-4403-8448-c61416c6341b\" (UID: \"b841d1a4-37c8-4403-8448-c61416c6341b\") " Nov 02 22:14:30.254280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.254259 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a70dac-c289-41e3-94f2-e522b78fd6c9-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.254363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.254287 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a70dac-c289-41e3-94f2-e522b78fd6c9-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.254363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.254304 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvcb\" (UniqueName: \"kubernetes.io/projected/92a70dac-c289-41e3-94f2-e522b78fd6c9-kube-api-access-8hvcb\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.255272 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.255083 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home" pod "b841d1a4-37c8-4403-8448-c61416c6341b" (UID: "b841d1a4-37c8-4403-8448-c61416c6341b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.255625 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.255589 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache" pod "b841d1a4-37c8-4403-8448-c61416c6341b" (UID: "b841d1a4-37c8-4403-8448-c61416c6341b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.256120 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.256099 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir" pod "b841d1a4-37c8-4403-8448-c61416c6341b" (UID: "b841d1a4-37c8-4403-8448-c61416c6341b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.257301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.257275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs" pod "b841d1a4-37c8-4403-8448-c61416c6341b" (UID: "b841d1a4-37c8-4403-8448-c61416c6341b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:30.257428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.257407 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn" pod "b841d1a4-37c8-4403-8448-c61416c6341b" (UID: "b841d1a4-37c8-4403-8448-c61416c6341b"). InnerVolumeSpecName "kube-api-access-d2mfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:30.258304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.258273 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm" pod "b841d1a4-37c8-4403-8448-c61416c6341b" (UID: "b841d1a4-37c8-4403-8448-c61416c6341b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.358658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.358579 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.358658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.358619 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.358658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.358635 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mfn\" (UniqueName: \"kubernetes.io/projected/b841d1a4-37c8-4403-8448-c61416c6341b-kube-api-access-d2mfn\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.358658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.358649 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.358918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.358664 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b841d1a4-37c8-4403-8448-c61416c6341b-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.358918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.358696 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b841d1a4-37c8-4403-8448-c61416c6341b-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.361096 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361065 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn"] Nov 02 22:14:30.361885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361859 5091 cpu_manager.go:414] "containerMap: removing container" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" Nov 02 22:14:30.361885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361883 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361894 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361900 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361913 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361919 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361931 5091 cpu_manager.go:414] "containerMap: removing container" podUID="eb19f9ba-a147-4d83-9edf-99c569468d3e" containerName="revoke-keys" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361937 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb19f9ba-a147-4d83-9edf-99c569468d3e" containerName="revoke-keys" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361968 5091 cpu_manager.go:414] "containerMap: removing container" podUID="01e124af-a6ec-4c9f-8a4f-7cb419ad7340" containerName="revoke-keys" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361977 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e124af-a6ec-4c9f-8a4f-7cb419ad7340" containerName="revoke-keys" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361986 5091 cpu_manager.go:414] "containerMap: removing container" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.361991 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362008 5091 cpu_manager.go:414] "containerMap: removing container" podUID="66263ea9-bb12-4968-83a6-ffc87473abb8" containerName="revoke-keys" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362013 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66263ea9-bb12-4968-83a6-ffc87473abb8" containerName="revoke-keys" Nov 02 22:14:30.362111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362105 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="66263ea9-bb12-4968-83a6-ffc87473abb8" containerName="revoke-keys" Nov 02 22:14:30.362806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362123 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerName="main" Nov 02 22:14:30.362806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362130 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" containerName="main" Nov 02 22:14:30.362806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362137 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7896c66-d594-421a-8c95-91559ccbea35" containerName="main" Nov 02 22:14:30.362806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362143 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb19f9ba-a147-4d83-9edf-99c569468d3e" containerName="revoke-keys" Nov 02 22:14:30.362806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362152 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e124af-a6ec-4c9f-8a4f-7cb419ad7340" containerName="revoke-keys" Nov 02 22:14:30.362806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.362158 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="146b8ca0-a44d-4f48-a022-a4d335e92135" containerName="main" Nov 02 22:14:30.366860 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.366841 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.420606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.420578 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn"] Nov 02 22:14:30.460582 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.460544 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.460747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.460727 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.460789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.460750 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtjn\" (UniqueName: \"kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.485067 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.485041 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:30.485492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.485064 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:14:30.485492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.485135 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:30.485492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.485434 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:30.497852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.497829 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5"] Nov 02 22:14:30.501925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.501911 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.559018 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.558989 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5"] Nov 02 22:14:30.562098 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.562071 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75w29\" (UniqueName: \"kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.562228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.562190 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.562228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.562214 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtjn\" (UniqueName: \"kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.562359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.562239 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.562359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.562299 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.562474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.562441 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.563038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.563016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.564900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.564873 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.619102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.619078 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtjn\" (UniqueName: \"kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn\") pod \"maas-api-revoke-sub-1ed4a824-6s9xn\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.663256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.663227 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75w29\" (UniqueName: \"kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.663375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.663326 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.663417 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.663376 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.664105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.664086 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.665577 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.665559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.692898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.692875 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:30.719300 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.719278 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75w29\" (UniqueName: \"kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29\") pod \"maas-api-revoke-sub-dae7283f-2npd5\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.777979 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.777947 5091 generic.go:356] "Generic (PLEG): container finished" podID="b841d1a4-37c8-4403-8448-c61416c6341b" containerID="bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229" exitCode=0 Nov 02 22:14:30.778279 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.778207 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" event={"ID":"b841d1a4-37c8-4403-8448-c61416c6341b","Type":"ContainerDied","Data":"bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229"} Nov 02 22:14:30.778777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.778297 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" Nov 02 22:14:30.778777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.778358 5091 scope.go:122] "RemoveContainer" containerID="bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229" Nov 02 22:14:30.779414 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.779349 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9" event={"ID":"b841d1a4-37c8-4403-8448-c61416c6341b","Type":"ContainerDied","Data":"df7e279122b00848fa2d1e8cf5217e816916df3f7d7caacd6878a40f536daebe"} Nov 02 22:14:30.781970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.781691 5091 generic.go:356] "Generic (PLEG): container finished" podID="92a70dac-c289-41e3-94f2-e522b78fd6c9" containerID="69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007" exitCode=0 Nov 02 22:14:30.781970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.781745 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" Nov 02 22:14:30.781970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.781770 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" event={"ID":"92a70dac-c289-41e3-94f2-e522b78fd6c9","Type":"ContainerDied","Data":"69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007"} Nov 02 22:14:30.781970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.781799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw" event={"ID":"92a70dac-c289-41e3-94f2-e522b78fd6c9","Type":"ContainerDied","Data":"c12a8e4fd1435849d08e9d7e9f55953008a38c3dad136a7949834eb60f3fc234"} Nov 02 22:14:30.797504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.797477 5091 scope.go:122] "RemoveContainer" containerID="bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229" Nov 02 22:14:30.797833 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:30.797806 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229\": container with ID starting with bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229 not found: ID does not exist" containerID="bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229" Nov 02 22:14:30.797951 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.797839 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229"} err="failed to get container status \"bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229\": rpc error: code = NotFound desc = could not find container \"bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229\": container with ID starting with bf1719374ff1770494150ff1c3a921628767c3773e3fbad16e7b810d9c8aa229 not found: ID does not exist" Nov 02 22:14:30.797951 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.797868 5091 scope.go:122] "RemoveContainer" containerID="69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007" Nov 02 22:14:30.808956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.808767 5091 scope.go:122] "RemoveContainer" containerID="69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007" Nov 02 22:14:30.809128 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:30.809097 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007\": container with ID starting with 69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007 not found: ID does not exist" containerID="69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007" Nov 02 22:14:30.809170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.809135 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007"} err="failed to get container status \"69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007\": rpc error: code = NotFound desc = could not find container \"69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007\": container with ID starting with 69c0de88dc8245db5c6fef1242ff74360b879c20ce7f1fd4e799d7711b2d7007 not found: ID does not exist" Nov 02 22:14:30.816360 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.816335 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:30.943464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.943432 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw"] Nov 02 22:14:30.948932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.948903 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-2-w3-64438f-kserve-6d66cc68d-t2cjw"] Nov 02 22:14:30.985017 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:30.984989 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a70dac-c289-41e3-94f2-e522b78fd6c9" path="/var/lib/kubelet/pods/92a70dac-c289-41e3-94f2-e522b78fd6c9/volumes" Nov 02 22:14:31.056439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.056401 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9"] Nov 02 22:14:31.061129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.061099 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/simulator-w3-64438f-kserve-dbfc88845-6mtv9"] Nov 02 22:14:31.068562 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.068537 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn"] Nov 02 22:14:31.069803 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:31.069777 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9390ded7_fee7_4f13_853c_b849b94ba11e.slice/crio-412d02c34989a7da6b3d21a6f54aed5aa0ab2900365deea941a928fccfc85ed0 WatchSource:0}: Error finding container 412d02c34989a7da6b3d21a6f54aed5aa0ab2900365deea941a928fccfc85ed0: Status 404 returned error can't find the container with id 412d02c34989a7da6b3d21a6f54aed5aa0ab2900365deea941a928fccfc85ed0 Nov 02 22:14:31.205654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.205447 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5"] Nov 02 22:14:31.206429 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:31.206382 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b53642_1053_4cb4_9955_82ea50cc2d28.slice/crio-679ea72a0ce45bbc8bbb9444c8f1c502de470f3c2ea69c65d9548f1227ac0147 WatchSource:0}: Error finding container 679ea72a0ce45bbc8bbb9444c8f1c502de470f3c2ea69c65d9548f1227ac0147: Status 404 returned error can't find the container with id 679ea72a0ce45bbc8bbb9444c8f1c502de470f3c2ea69c65d9548f1227ac0147 Nov 02 22:14:31.667432 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.667406 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 22:14:31.780049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.779967 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780015 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780130 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780205 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780254 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780317 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780553 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780368 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780553 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780452 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token\") pod \"b9e68f91-e56f-41f1-a548-68bf84431876\" (UID: \"b9e68f91-e56f-41f1-a548-68bf84431876\") " Nov 02 22:14:31.780806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780781 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:31.780885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780853 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.780954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.780878 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.781071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.781054 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.781142 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.781069 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.781215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.781195 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e68f91-e56f-41f1-a548-68bf84431876-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.781280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.781219 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.781280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.781234 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.784028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.783324 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "kube-api-access-kj92b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:31.784255 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.784226 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.784504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.784478 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:31.789997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.789963 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo" pod "b9e68f91-e56f-41f1-a548-68bf84431876" (UID: "b9e68f91-e56f-41f1-a548-68bf84431876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:14:31.799464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.799431 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" event={"ID":"f0b53642-1053-4cb4-9955-82ea50cc2d28","Type":"ContainerStarted","Data":"cf21465d4acbcc314a166e5fcbf1fcff70e9843a949e75b12f7719157289c2e7"} Nov 02 22:14:31.799631 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.799493 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" event={"ID":"f0b53642-1053-4cb4-9955-82ea50cc2d28","Type":"ContainerStarted","Data":"679ea72a0ce45bbc8bbb9444c8f1c502de470f3c2ea69c65d9548f1227ac0147"} Nov 02 22:14:31.806919 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.806891 5091 generic.go:356] "Generic (PLEG): container finished" podID="b9e68f91-e56f-41f1-a548-68bf84431876" containerID="2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d" exitCode=0 Nov 02 22:14:31.807012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.806988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" event={"ID":"b9e68f91-e56f-41f1-a548-68bf84431876","Type":"ContainerDied","Data":"2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d"} Nov 02 22:14:31.807089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.807036 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" event={"ID":"b9e68f91-e56f-41f1-a548-68bf84431876","Type":"ContainerDied","Data":"a4defb87200c307837bf6d452d31715c800d791da9219f29e688f6018203e5c3"} Nov 02 22:14:31.807089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.806992 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h" Nov 02 22:14:31.807089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.807068 5091 scope.go:122] "RemoveContainer" containerID="2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d" Nov 02 22:14:31.812055 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.811280 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" event={"ID":"9390ded7-fee7-4f13-853c-b849b94ba11e","Type":"ContainerStarted","Data":"4990c26dcdeef26d618bf2d8866c2f1dd5a342bd4b1adb54bc632b68c25f2221"} Nov 02 22:14:31.812055 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.811330 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" event={"ID":"9390ded7-fee7-4f13-853c-b849b94ba11e","Type":"ContainerStarted","Data":"412d02c34989a7da6b3d21a6f54aed5aa0ab2900365deea941a928fccfc85ed0"} Nov 02 22:14:31.820738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.820650 5091 scope.go:122] "RemoveContainer" containerID="2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d" Nov 02 22:14:31.820993 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:31.820963 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d\": container with ID starting with 2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d not found: ID does not exist" containerID="2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d" Nov 02 22:14:31.821095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.821001 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d"} err="failed to get container status \"2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d\": rpc error: code = NotFound desc = could not find container \"2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d\": container with ID starting with 2b59b9d5a682bb6f551d11d7b1aef8bbf986ae7133a261b1ff183784dfed713d not found: ID does not exist" Nov 02 22:14:31.882638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.882614 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.882638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.882638 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.882820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.882664 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.882820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.882704 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj92b\" (UniqueName: \"kubernetes.io/projected/b9e68f91-e56f-41f1-a548-68bf84431876-kube-api-access-kj92b\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.882820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.882721 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e68f91-e56f-41f1-a548-68bf84431876-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.882820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.882738 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e68f91-e56f-41f1-a548-68bf84431876-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.912885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:31.912843 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" podStartSLOduration=1.9128307850000001 podStartE2EDuration="1.912830785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:31.910296788 +0000 UTC m=+2419.072392007" watchObservedRunningTime="2026-11-02 22:14:31.912830785 +0000 UTC m=+2419.074925905" Nov 02 22:14:32.024062 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.024027 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h"] Nov 02 22:14:32.027760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.027737 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-xsn9h"] Nov 02 22:14:32.136336 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.136291 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" podStartSLOduration=2.136271129 podStartE2EDuration="2.136271129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:32.133304105 +0000 UTC m=+2419.295399224" watchObservedRunningTime="2026-11-02 22:14:32.136271129 +0000 UTC m=+2419.298366250" Nov 02 22:14:32.820588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.820553 5091 generic.go:356] "Generic (PLEG): container finished" podID="f0b53642-1053-4cb4-9955-82ea50cc2d28" containerID="cf21465d4acbcc314a166e5fcbf1fcff70e9843a949e75b12f7719157289c2e7" exitCode=0 Nov 02 22:14:32.821095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.820634 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" event={"ID":"f0b53642-1053-4cb4-9955-82ea50cc2d28","Type":"ContainerDied","Data":"cf21465d4acbcc314a166e5fcbf1fcff70e9843a949e75b12f7719157289c2e7"} Nov 02 22:14:32.823614 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.823587 5091 generic.go:356] "Generic (PLEG): container finished" podID="9390ded7-fee7-4f13-853c-b849b94ba11e" containerID="4990c26dcdeef26d618bf2d8866c2f1dd5a342bd4b1adb54bc632b68c25f2221" exitCode=0 Nov 02 22:14:32.823748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.823665 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" event={"ID":"9390ded7-fee7-4f13-853c-b849b94ba11e","Type":"ContainerDied","Data":"4990c26dcdeef26d618bf2d8866c2f1dd5a342bd4b1adb54bc632b68c25f2221"} Nov 02 22:14:32.977325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.977294 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b841d1a4-37c8-4403-8448-c61416c6341b" path="/var/lib/kubelet/pods/b841d1a4-37c8-4403-8448-c61416c6341b/volumes" Nov 02 22:14:32.977846 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:32.977831 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" path="/var/lib/kubelet/pods/b9e68f91-e56f-41f1-a548-68bf84431876/volumes" Nov 02 22:14:33.772494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.772460 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6"] Nov 02 22:14:33.773129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.773108 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" Nov 02 22:14:33.773129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.773129 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" Nov 02 22:14:33.773332 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.773318 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e68f91-e56f-41f1-a548-68bf84431876" containerName="istio-proxy" Nov 02 22:14:33.777982 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.777961 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:33.862221 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.861569 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6"] Nov 02 22:14:33.919249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.919215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:33.919545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.919519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:33.919753 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:33.919733 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mnd\" (UniqueName: \"kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.020996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.020959 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.021181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.021046 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.021181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.021119 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mnd\" (UniqueName: \"kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.022797 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.022738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.026115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.026086 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.059197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.059173 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:34.079647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.079622 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mnd\" (UniqueName: \"kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.091601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.091578 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:34.097727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.097707 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:34.121718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.121692 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca\") pod \"9390ded7-fee7-4f13-853c-b849b94ba11e\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " Nov 02 22:14:34.121896 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.121780 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca\") pod \"9390ded7-fee7-4f13-853c-b849b94ba11e\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " Nov 02 22:14:34.121896 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.121855 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn\" (UniqueName: \"kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn\") pod \"9390ded7-fee7-4f13-853c-b849b94ba11e\" (UID: \"9390ded7-fee7-4f13-853c-b849b94ba11e\") " Nov 02 22:14:34.122815 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.122524 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca" pod "9390ded7-fee7-4f13-853c-b849b94ba11e" (UID: "9390ded7-fee7-4f13-853c-b849b94ba11e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:34.124685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.124649 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca" pod "9390ded7-fee7-4f13-853c-b849b94ba11e" (UID: "9390ded7-fee7-4f13-853c-b849b94ba11e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:34.124833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.124808 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn" pod "9390ded7-fee7-4f13-853c-b849b94ba11e" (UID: "9390ded7-fee7-4f13-853c-b849b94ba11e"). InnerVolumeSpecName "kube-api-access-pxtjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.223822 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca\") pod \"f0b53642-1053-4cb4-9955-82ea50cc2d28\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.223930 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29\" (UniqueName: \"kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29\") pod \"f0b53642-1053-4cb4-9955-82ea50cc2d28\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.224081 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca\") pod \"f0b53642-1053-4cb4-9955-82ea50cc2d28\" (UID: \"f0b53642-1053-4cb4-9955-82ea50cc2d28\") " Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.224136 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca" pod "f0b53642-1053-4cb4-9955-82ea50cc2d28" (UID: "f0b53642-1053-4cb4-9955-82ea50cc2d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.224704 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.224727 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9390ded7-fee7-4f13-853c-b849b94ba11e-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.224743 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtjn\" (UniqueName: \"kubernetes.io/projected/9390ded7-fee7-4f13-853c-b849b94ba11e-kube-api-access-pxtjn\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:34.225690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.224757 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:34.232257 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.226280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29" pod "f0b53642-1053-4cb4-9955-82ea50cc2d28" (UID: "f0b53642-1053-4cb4-9955-82ea50cc2d28"). InnerVolumeSpecName "kube-api-access-75w29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:34.232257 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.226917 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca" pod "f0b53642-1053-4cb4-9955-82ea50cc2d28" (UID: "f0b53642-1053-4cb4-9955-82ea50cc2d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:34.329070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.328969 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b53642-1053-4cb4-9955-82ea50cc2d28-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:34.329070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.329006 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75w29\" (UniqueName: \"kubernetes.io/projected/f0b53642-1053-4cb4-9955-82ea50cc2d28-kube-api-access-75w29\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:34.482755 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.482722 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6"] Nov 02 22:14:34.483969 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:14:34.483940 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1866a79a_b747_4d9f_95fc_2edd8ce1b801.slice/crio-3f30d0c87329310b322843181ee91e3de61c34a11b5c0e9ad8bd391c6fa1ab8c WatchSource:0}: Error finding container 3f30d0c87329310b322843181ee91e3de61c34a11b5c0e9ad8bd391c6fa1ab8c: Status 404 returned error can't find the container with id 3f30d0c87329310b322843181ee91e3de61c34a11b5c0e9ad8bd391c6fa1ab8c Nov 02 22:14:34.853663 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.853634 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" Nov 02 22:14:34.853663 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.853652 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5" event={"ID":"f0b53642-1053-4cb4-9955-82ea50cc2d28","Type":"ContainerDied","Data":"679ea72a0ce45bbc8bbb9444c8f1c502de470f3c2ea69c65d9548f1227ac0147"} Nov 02 22:14:34.853945 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.853724 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679ea72a0ce45bbc8bbb9444c8f1c502de470f3c2ea69c65d9548f1227ac0147" Nov 02 22:14:34.855487 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.855457 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" event={"ID":"1866a79a-b747-4d9f-95fc-2edd8ce1b801","Type":"ContainerStarted","Data":"94d313b83d2d36b5306bfd87171ea085de9b86a12c1c05d05d8dd916c6f3c0d8"} Nov 02 22:14:34.855616 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.855500 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" event={"ID":"1866a79a-b747-4d9f-95fc-2edd8ce1b801","Type":"ContainerStarted","Data":"3f30d0c87329310b322843181ee91e3de61c34a11b5c0e9ad8bd391c6fa1ab8c"} Nov 02 22:14:34.856910 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.856894 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" Nov 02 22:14:34.857011 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.856942 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn" event={"ID":"9390ded7-fee7-4f13-853c-b849b94ba11e","Type":"ContainerDied","Data":"412d02c34989a7da6b3d21a6f54aed5aa0ab2900365deea941a928fccfc85ed0"} Nov 02 22:14:34.857011 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.856978 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412d02c34989a7da6b3d21a6f54aed5aa0ab2900365deea941a928fccfc85ed0" Nov 02 22:14:34.968360 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:34.968271 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" podStartSLOduration=1.9682564500000002 podStartE2EDuration="1.96825645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:34.96681361 +0000 UTC m=+2422.128908731" watchObservedRunningTime="2026-11-02 22:14:34.96825645 +0000 UTC m=+2422.130351549" Nov 02 22:14:35.691311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:35.691261 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn"] Nov 02 22:14:35.779960 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:35.779931 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5"] Nov 02 22:14:35.867825 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:35.867793 5091 generic.go:356] "Generic (PLEG): container finished" podID="1866a79a-b747-4d9f-95fc-2edd8ce1b801" containerID="94d313b83d2d36b5306bfd87171ea085de9b86a12c1c05d05d8dd916c6f3c0d8" exitCode=0 Nov 02 22:14:35.868007 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:35.867876 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" event={"ID":"1866a79a-b747-4d9f-95fc-2edd8ce1b801","Type":"ContainerDied","Data":"94d313b83d2d36b5306bfd87171ea085de9b86a12c1c05d05d8dd916c6f3c0d8"} Nov 02 22:14:37.224545 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:37.213401 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:14:37.647849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:37.647798 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:37.964721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:37.964613 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" event={"ID":"1866a79a-b747-4d9f-95fc-2edd8ce1b801","Type":"ContainerDied","Data":"3f30d0c87329310b322843181ee91e3de61c34a11b5c0e9ad8bd391c6fa1ab8c"} Nov 02 22:14:37.965022 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:37.965000 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f30d0c87329310b322843181ee91e3de61c34a11b5c0e9ad8bd391c6fa1ab8c" Nov 02 22:14:37.990702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:37.985848 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:38.085002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.084968 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca\") pod \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " Nov 02 22:14:38.085354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.085332 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca\") pod \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " Nov 02 22:14:38.085665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.085645 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd\" (UniqueName: \"kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd\") pod \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\" (UID: \"1866a79a-b747-4d9f-95fc-2edd8ce1b801\") " Nov 02 22:14:38.097282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.097250 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca" pod "1866a79a-b747-4d9f-95fc-2edd8ce1b801" (UID: "1866a79a-b747-4d9f-95fc-2edd8ce1b801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:38.097606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.097584 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd" pod "1866a79a-b747-4d9f-95fc-2edd8ce1b801" (UID: "1866a79a-b747-4d9f-95fc-2edd8ce1b801"). InnerVolumeSpecName "kube-api-access-45mnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:38.130708 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.129721 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca" pod "1866a79a-b747-4d9f-95fc-2edd8ce1b801" (UID: "1866a79a-b747-4d9f-95fc-2edd8ce1b801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:38.188159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.188127 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mnd\" (UniqueName: \"kubernetes.io/projected/1866a79a-b747-4d9f-95fc-2edd8ce1b801-kube-api-access-45mnd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.191751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.191721 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.191927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.191911 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1866a79a-b747-4d9f-95fc-2edd8ce1b801-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.976940 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:38.976858 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6" Nov 02 22:14:39.475998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:39.475958 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:40.693136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:40.693093 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerName="payload-processing" containerID="cri-o://7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825" gracePeriod=25 Nov 02 22:14:40.780955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:40.780913 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerName="payload-pre-processing" containerID="cri-o://baf3ad22e0f711b04be34b0d490943a9b8a3f15e79f4bccd38bc8da02ce15de0" gracePeriod=25 Nov 02 22:14:41.957115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:41.957091 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 22:14:42.000702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.000639 5091 generic.go:356] "Generic (PLEG): container finished" podID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerID="f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff" exitCode=0 Nov 02 22:14:42.000702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.000700 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" Nov 02 22:14:42.000913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.000705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" event={"ID":"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1","Type":"ContainerDied","Data":"f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff"} Nov 02 22:14:42.000913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.000764 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh" event={"ID":"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1","Type":"ContainerDied","Data":"d3b2fb9eb9801df4d8092d87ee3fd480868da24e0bde5d73f63aa9472b56a297"} Nov 02 22:14:42.000913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.000787 5091 scope.go:122] "RemoveContainer" containerID="f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff" Nov 02 22:14:42.011115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.011098 5091 scope.go:122] "RemoveContainer" containerID="f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff" Nov 02 22:14:42.011388 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:42.011369 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff\": container with ID starting with f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff not found: ID does not exist" containerID="f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff" Nov 02 22:14:42.011458 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.011396 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff"} err="failed to get container status \"f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff\": rpc error: code = NotFound desc = could not find container \"f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff\": container with ID starting with f157585c1aa5549f73fc47fc450b32cf37826193c700ad790fcefc9eb3d606ff not found: ID does not exist" Nov 02 22:14:42.142076 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.142042 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume\") pod \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " Nov 02 22:14:42.142246 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.142214 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7\" (UniqueName: \"kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7\") pod \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\" (UID: \"cafc7e1c-50b3-4018-81fa-3da2fc5c04f1\") " Nov 02 22:14:42.142406 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.142380 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume" pod "cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" (UID: "cafc7e1c-50b3-4018-81fa-3da2fc5c04f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:42.142615 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.142602 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.144332 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.144306 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7" pod "cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" (UID: "cafc7e1c-50b3-4018-81fa-3da2fc5c04f1"). InnerVolumeSpecName "kube-api-access-b49c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:42.243430 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.243359 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b49c7\" (UniqueName: \"kubernetes.io/projected/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1-kube-api-access-b49c7\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.421500 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.421454 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh"] Nov 02 22:14:42.424804 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.424779 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb8112-75b95748bc-6prgh"] Nov 02 22:14:42.486478 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.486436 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" containerID="cri-o://c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956" gracePeriod=45 Nov 02 22:14:42.500147 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.500081 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" containerID="cri-o://cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f" gracePeriod=45 Nov 02 22:14:42.774978 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.774910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 22:14:42.803752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.803727 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958410 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir\") pod \"4a8e5011-11f6-4298-bc22-4f03b28ee856\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958454 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn\" (UniqueName: \"kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn\") pod \"764023c0-d67a-451d-a0ec-56281fd4cfc3\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958539 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache\") pod \"764023c0-d67a-451d-a0ec-56281fd4cfc3\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958570 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache\") pod \"4a8e5011-11f6-4298-bc22-4f03b28ee856\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958590 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs\") pod \"4a8e5011-11f6-4298-bc22-4f03b28ee856\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958634 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home\") pod \"4a8e5011-11f6-4298-bc22-4f03b28ee856\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958661 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm\") pod \"764023c0-d67a-451d-a0ec-56281fd4cfc3\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958748 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4\" (UniqueName: \"kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4\") pod \"4a8e5011-11f6-4298-bc22-4f03b28ee856\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958814 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs\" (UniqueName: \"kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs\") pod \"764023c0-d67a-451d-a0ec-56281fd4cfc3\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958850 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home\") pod \"764023c0-d67a-451d-a0ec-56281fd4cfc3\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958891 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir\") pod \"764023c0-d67a-451d-a0ec-56281fd4cfc3\" (UID: \"764023c0-d67a-451d-a0ec-56281fd4cfc3\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.958968 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm\") pod \"4a8e5011-11f6-4298-bc22-4f03b28ee856\" (UID: \"4a8e5011-11f6-4298-bc22-4f03b28ee856\") " Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.959265 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir" pod "4a8e5011-11f6-4298-bc22-4f03b28ee856" (UID: "4a8e5011-11f6-4298-bc22-4f03b28ee856"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.959271 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache" pod "764023c0-d67a-451d-a0ec-56281fd4cfc3" (UID: "764023c0-d67a-451d-a0ec-56281fd4cfc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.959505 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home" pod "764023c0-d67a-451d-a0ec-56281fd4cfc3" (UID: "764023c0-d67a-451d-a0ec-56281fd4cfc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.961468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.959716 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir" pod "764023c0-d67a-451d-a0ec-56281fd4cfc3" (UID: "764023c0-d67a-451d-a0ec-56281fd4cfc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.961827 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache" pod "4a8e5011-11f6-4298-bc22-4f03b28ee856" (UID: "4a8e5011-11f6-4298-bc22-4f03b28ee856"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.961962 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.961988 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962008 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962022 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962092 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm" pod "764023c0-d67a-451d-a0ec-56281fd4cfc3" (UID: "764023c0-d67a-451d-a0ec-56281fd4cfc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962142 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm" pod "4a8e5011-11f6-4298-bc22-4f03b28ee856" (UID: "4a8e5011-11f6-4298-bc22-4f03b28ee856"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962284 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs" pod "764023c0-d67a-451d-a0ec-56281fd4cfc3" (UID: "764023c0-d67a-451d-a0ec-56281fd4cfc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:42.962523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962367 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn" pod "764023c0-d67a-451d-a0ec-56281fd4cfc3" (UID: "764023c0-d67a-451d-a0ec-56281fd4cfc3"). InnerVolumeSpecName "kube-api-access-xnkxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:42.962833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.962801 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home" pod "4a8e5011-11f6-4298-bc22-4f03b28ee856" (UID: "4a8e5011-11f6-4298-bc22-4f03b28ee856"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:42.963369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.963349 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs" pod "4a8e5011-11f6-4298-bc22-4f03b28ee856" (UID: "4a8e5011-11f6-4298-bc22-4f03b28ee856"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:42.963717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.963664 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4" pod "4a8e5011-11f6-4298-bc22-4f03b28ee856" (UID: "4a8e5011-11f6-4298-bc22-4f03b28ee856"). InnerVolumeSpecName "kube-api-access-ngjc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:42.977080 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:42.977054 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" path="/var/lib/kubelet/pods/cafc7e1c-50b3-4018-81fa-3da2fc5c04f1/volumes" Nov 02 22:14:43.010842 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.010815 5091 generic.go:356] "Generic (PLEG): container finished" podID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerID="cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f" exitCode=0 Nov 02 22:14:43.010987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.010894 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" Nov 02 22:14:43.010987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.010897 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" event={"ID":"4a8e5011-11f6-4298-bc22-4f03b28ee856","Type":"ContainerDied","Data":"cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f"} Nov 02 22:14:43.011079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.011018 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" event={"ID":"4a8e5011-11f6-4298-bc22-4f03b28ee856","Type":"ContainerDied","Data":"0c5d7722ab209dd63ab358ba2ea0f24379455a36fb9dbd0168d607cd8b7f193c"} Nov 02 22:14:43.011079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.011056 5091 scope.go:122] "RemoveContainer" containerID="cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f" Nov 02 22:14:43.012986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.012619 5091 generic.go:356] "Generic (PLEG): container finished" podID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerID="c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956" exitCode=0 Nov 02 22:14:43.012986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.012729 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" event={"ID":"764023c0-d67a-451d-a0ec-56281fd4cfc3","Type":"ContainerDied","Data":"c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956"} Nov 02 22:14:43.012986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.012748 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" Nov 02 22:14:43.012986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.012768 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs" event={"ID":"764023c0-d67a-451d-a0ec-56281fd4cfc3","Type":"ContainerDied","Data":"e2055c5dd9a073c78f64fdd01a22a12bf2a568e533a006ad2ccbc27630e9092c"} Nov 02 22:14:43.026014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.025993 5091 scope.go:122] "RemoveContainer" containerID="cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f" Nov 02 22:14:43.026402 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:43.026370 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f\": container with ID starting with cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f not found: ID does not exist" containerID="cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f" Nov 02 22:14:43.026479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.026409 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f"} err="failed to get container status \"cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f\": rpc error: code = NotFound desc = could not find container \"cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f\": container with ID starting with cec91a29282729bfeb8263f6696608c9e0aa64fe2ae39f10067d016ce161ac7f not found: ID does not exist" Nov 02 22:14:43.026479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.026428 5091 scope.go:122] "RemoveContainer" containerID="c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956" Nov 02 22:14:43.038921 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.038898 5091 scope.go:122] "RemoveContainer" containerID="c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956" Nov 02 22:14:43.039290 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:14:43.039263 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956\": container with ID starting with c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956 not found: ID does not exist" containerID="c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956" Nov 02 22:14:43.039357 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.039300 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956"} err="failed to get container status \"c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956\": rpc error: code = NotFound desc = could not find container \"c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956\": container with ID starting with c6d0c2a79b47cd3255a93f93dcfdd2d9584447dfec231f968880620cb5954956 not found: ID does not exist" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.062934 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.062970 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a8e5011-11f6-4298-bc22-4f03b28ee856-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.062985 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.062999 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764023c0-d67a-451d-a0ec-56281fd4cfc3-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.063014 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngjc4\" (UniqueName: \"kubernetes.io/projected/4a8e5011-11f6-4298-bc22-4f03b28ee856-kube-api-access-ngjc4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.063029 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764023c0-d67a-451d-a0ec-56281fd4cfc3-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.063044 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a8e5011-11f6-4298-bc22-4f03b28ee856-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.063081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.063058 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnkxn\" (UniqueName: \"kubernetes.io/projected/764023c0-d67a-451d-a0ec-56281fd4cfc3-kube-api-access-xnkxn\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:43.174422 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.174382 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd"] Nov 02 22:14:43.177698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.177663 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd"] Nov 02 22:14:43.285469 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.285380 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs"] Nov 02 22:14:43.289460 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.289434 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ffe386/premium-w1-ffe386-kserve-6768f9f5c5-mckbs"] Nov 02 22:14:43.559960 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.559872 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:43.560123 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.559969 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:43.751587 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:43.751547 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ffe386/simulator-w1-ffe386-kserve-f9fb85db6-4mmvd" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 22:14:44.977637 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:44.977603 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" path="/var/lib/kubelet/pods/4a8e5011-11f6-4298-bc22-4f03b28ee856/volumes" Nov 02 22:14:44.978134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:44.978118 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" path="/var/lib/kubelet/pods/764023c0-d67a-451d-a0ec-56281fd4cfc3/volumes" Nov 02 22:14:47.062489 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.062460 5091 generic.go:356] "Generic (PLEG): container finished" podID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerID="46407fd0f8a8bd4deeed4c62a6d10ea625fa4f8c655464d729545ec65e60b3ba" exitCode=0 Nov 02 22:14:47.063022 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.062996 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" event={"ID":"265d1fb0-e6b0-4964-88f9-bdb1007447f5","Type":"ContainerDied","Data":"46407fd0f8a8bd4deeed4c62a6d10ea625fa4f8c655464d729545ec65e60b3ba"} Nov 02 22:14:47.144967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.144928 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 22:14:47.229195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.229162 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume\") pod \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " Nov 02 22:14:47.229580 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.229556 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume" pod "265d1fb0-e6b0-4964-88f9-bdb1007447f5" (UID: "265d1fb0-e6b0-4964-88f9-bdb1007447f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:47.330581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.330541 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5\" (UniqueName: \"kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5\") pod \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\" (UID: \"265d1fb0-e6b0-4964-88f9-bdb1007447f5\") " Nov 02 22:14:47.330860 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.330842 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265d1fb0-e6b0-4964-88f9-bdb1007447f5-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:47.332903 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.332871 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5" pod "265d1fb0-e6b0-4964-88f9-bdb1007447f5" (UID: "265d1fb0-e6b0-4964-88f9-bdb1007447f5"). InnerVolumeSpecName "kube-api-access-8nsm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:47.431932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:47.431896 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nsm5\" (UniqueName: \"kubernetes.io/projected/265d1fb0-e6b0-4964-88f9-bdb1007447f5-kube-api-access-8nsm5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:48.072277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:48.072245 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" Nov 02 22:14:48.072812 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:48.072282 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f" event={"ID":"265d1fb0-e6b0-4964-88f9-bdb1007447f5","Type":"ContainerDied","Data":"a3444796168f2d92518524b3ac7fd022ca1b79e21a86da3ae4af50cb7911c8ab"} Nov 02 22:14:48.072812 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:48.072339 5091 scope.go:122] "RemoveContainer" containerID="46407fd0f8a8bd4deeed4c62a6d10ea625fa4f8c655464d729545ec65e60b3ba" Nov 02 22:14:48.235087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:48.235048 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f"] Nov 02 22:14:48.284087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:48.284059 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-64438f-cb5564875-z785f"] Nov 02 22:14:48.978095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:48.978062 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" path="/var/lib/kubelet/pods/265d1fb0-e6b0-4964-88f9-bdb1007447f5/volumes" Nov 02 22:14:53.559196 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:53.559152 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:53.559659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:14:53.559170 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:15:00.165655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.165617 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv"] Nov 02 22:15:00.166493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166471 5091 cpu_manager.go:414] "containerMap: removing container" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" Nov 02 22:15:00.166545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166497 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" Nov 02 22:15:00.166545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166515 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" Nov 02 22:15:00.166545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166522 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166556 5091 cpu_manager.go:414] "containerMap: removing container" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerName="payload-processing" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166564 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerName="payload-processing" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166576 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f0b53642-1053-4cb4-9955-82ea50cc2d28" containerName="revoke-keys" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166583 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b53642-1053-4cb4-9955-82ea50cc2d28" containerName="revoke-keys" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166593 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1866a79a-b747-4d9f-95fc-2edd8ce1b801" containerName="revoke-keys" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166600 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1866a79a-b747-4d9f-95fc-2edd8ce1b801" containerName="revoke-keys" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166636 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9390ded7-fee7-4f13-853c-b849b94ba11e" containerName="revoke-keys" Nov 02 22:15:00.166647 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166644 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9390ded7-fee7-4f13-853c-b849b94ba11e" containerName="revoke-keys" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166693 5091 cpu_manager.go:414] "containerMap: removing container" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerName="payload-pre-processing" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166701 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerName="payload-pre-processing" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166845 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafc7e1c-50b3-4018-81fa-3da2fc5c04f1" containerName="payload-pre-processing" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166857 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="764023c0-d67a-451d-a0ec-56281fd4cfc3" containerName="main" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166866 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8e5011-11f6-4298-bc22-4f03b28ee856" containerName="main" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166879 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="265d1fb0-e6b0-4964-88f9-bdb1007447f5" containerName="payload-processing" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166887 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b53642-1053-4cb4-9955-82ea50cc2d28" containerName="revoke-keys" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166894 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1866a79a-b747-4d9f-95fc-2edd8ce1b801" containerName="revoke-keys" Nov 02 22:15:00.166939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.166912 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9390ded7-fee7-4f13-853c-b849b94ba11e" containerName="revoke-keys" Nov 02 22:15:00.174856 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.172780 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm"] Nov 02 22:15:00.176920 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.175108 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.179882 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.179847 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k"] Nov 02 22:15:00.183085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.183063 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.185998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.185976 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.215268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.215242 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh"] Nov 02 22:15:00.219099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.219080 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2"] Nov 02 22:15:00.219250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.219193 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.223415 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.223397 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2"] Nov 02 22:15:00.223504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.223491 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.227351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.227329 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.227657 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.227641 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz"] Nov 02 22:15:00.232132 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.232116 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg"] Nov 02 22:15:00.232248 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.232232 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.235827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.235810 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.261579 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.261551 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:15:00.261579 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.261556 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:15:00.261761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.261591 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:15:00.267195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.267174 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm"] Nov 02 22:15:00.270453 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.270437 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv"] Nov 02 22:15:00.273283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.273265 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k"] Nov 02 22:15:00.289846 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.289824 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.289988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.289969 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.290033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290006 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.290074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290039 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxvv\" (UniqueName: \"kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.290145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290127 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.290184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290156 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.290184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wrr\" (UniqueName: \"kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.290253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290195 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.290253 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sk6h\" (UniqueName: \"kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.290323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290275 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.290323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290315 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llk4v\" (UniqueName: \"kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.290392 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.290343 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkq8\" (UniqueName: \"kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.326721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.320287 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2"] Nov 02 22:15:00.326721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.324706 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz"] Nov 02 22:15:00.377693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.377636 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh"] Nov 02 22:15:00.382454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.382411 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2"] Nov 02 22:15:00.387103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.387078 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg"] Nov 02 22:15:00.391185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391153 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkq8\" (UniqueName: \"kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.391401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391232 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.391401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391293 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.391401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391349 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r45g7\" (UniqueName: \"kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7\") pod \"maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.391603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391457 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7qc\" (UniqueName: \"kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.391603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391493 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.391603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391521 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.391603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391556 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391609 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxvv\" (UniqueName: \"kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391701 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391729 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wrr\" (UniqueName: \"kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391761 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391792 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sk6h\" (UniqueName: \"kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.391852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391823 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.392241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.391861 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llk4v\" (UniqueName: \"kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.393712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.393237 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.393712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.393515 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.393889 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.393723 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.393889 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.393729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.393889 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.393814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.394473 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.394432 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.449744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.449648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llk4v\" (UniqueName: \"kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.449885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.449860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkq8\" (UniqueName: \"kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.449937 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.449886 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wrr\" (UniqueName: \"kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.450055 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.450037 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sk6h\" (UniqueName: \"kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h\") pod \"maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.450338 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.450310 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.450482 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.450467 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxvv\" (UniqueName: \"kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.492705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.492651 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.492831 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.492730 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r45g7\" (UniqueName: \"kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7\") pod \"maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.492831 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.492798 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7qc\" (UniqueName: \"kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.492953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.492939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.493217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.493198 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.493493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.493476 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.495375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.495362 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:00.503703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.503660 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:00.510731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.510701 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:00.536891 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.536868 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:00.544555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.544517 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:00.552809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.550211 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r45g7\" (UniqueName: \"kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7\") pod \"maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.552809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.550974 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7qc\" (UniqueName: \"kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc\") pod \"maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.552809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.552546 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:00.568548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.568522 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:00.570515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.568879 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:00.888490 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:00.888432 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv"] Nov 02 22:15:00.889662 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:00.889637 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5af250_15a5_436a_ac4c_66516346482f.slice/crio-ce057a6cb8e670c24a97175e33d812ad012ba873df3a04f3ec781e6e811b6830 WatchSource:0}: Error finding container ce057a6cb8e670c24a97175e33d812ad012ba873df3a04f3ec781e6e811b6830: Status 404 returned error can't find the container with id ce057a6cb8e670c24a97175e33d812ad012ba873df3a04f3ec781e6e811b6830 Nov 02 22:15:01.079824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.079793 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 22:15:01.087511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.087480 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2"] Nov 02 22:15:01.093270 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:01.093237 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6944953f_a3de_4004_a4f3_9f748d66d1c5.slice/crio-4881264c95c8977450c4bfaccd3170a73cb49dd2033385e1d3e6e45d95933f12 WatchSource:0}: Error finding container 4881264c95c8977450c4bfaccd3170a73cb49dd2033385e1d3e6e45d95933f12: Status 404 returned error can't find the container with id 4881264c95c8977450c4bfaccd3170a73cb49dd2033385e1d3e6e45d95933f12 Nov 02 22:15:01.129081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.129051 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm"] Nov 02 22:15:01.176487 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.176453 5091 generic.go:356] "Generic (PLEG): container finished" podID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerID="7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825" exitCode=0 Nov 02 22:15:01.176834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.176727 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" event={"ID":"acb8ba51-d617-41ba-8c4c-9d939c3ce13e","Type":"ContainerDied","Data":"7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825"} Nov 02 22:15:01.176834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.176797 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" event={"ID":"acb8ba51-d617-41ba-8c4c-9d939c3ce13e","Type":"ContainerDied","Data":"a38db4ecf78641460d05151e58b13520e5f5711ab70b143eef9d340230f33aa0"} Nov 02 22:15:01.176914 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.176834 5091 scope.go:122] "RemoveContainer" containerID="7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825" Nov 02 22:15:01.177013 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.176997 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn" Nov 02 22:15:01.179533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.179459 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerStarted","Data":"ce057a6cb8e670c24a97175e33d812ad012ba873df3a04f3ec781e6e811b6830"} Nov 02 22:15:01.185772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.185304 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerStarted","Data":"747001f911648d2777afc1e802970dea9c03d7f919f0d78548061cb6574b5b8d"} Nov 02 22:15:01.187185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.187155 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerStarted","Data":"4881264c95c8977450c4bfaccd3170a73cb49dd2033385e1d3e6e45d95933f12"} Nov 02 22:15:01.187905 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.187882 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh"] Nov 02 22:15:01.189582 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.189558 5091 generic.go:356] "Generic (PLEG): container finished" podID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerID="baf3ad22e0f711b04be34b0d490943a9b8a3f15e79f4bccd38bc8da02ce15de0" exitCode=0 Nov 02 22:15:01.189693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.189615 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" event={"ID":"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf","Type":"ContainerDied","Data":"baf3ad22e0f711b04be34b0d490943a9b8a3f15e79f4bccd38bc8da02ce15de0"} Nov 02 22:15:01.212821 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.212795 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2"] Nov 02 22:15:01.216475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.216412 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume\") pod \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " Nov 02 22:15:01.216617 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.216482 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5\" (UniqueName: \"kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5\") pod \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\" (UID: \"acb8ba51-d617-41ba-8c4c-9d939c3ce13e\") " Nov 02 22:15:01.217211 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.217173 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume" pod "acb8ba51-d617-41ba-8c4c-9d939c3ce13e" (UID: "acb8ba51-d617-41ba-8c4c-9d939c3ce13e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:01.217827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.217802 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:01.238002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.237967 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k"] Nov 02 22:15:01.241597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.241544 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg"] Nov 02 22:15:01.249491 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.249419 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz"] Nov 02 22:15:01.263973 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.263945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5" pod "acb8ba51-d617-41ba-8c4c-9d939c3ce13e" (UID: "acb8ba51-d617-41ba-8c4c-9d939c3ce13e"). InnerVolumeSpecName "kube-api-access-djjk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:01.266441 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:01.266207 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c800d43_835a_4235_8bac_196c3eec8b94.slice/crio-66bb76393e96fb0922ea2ad79da92c8a4a814490d53764fb8ff1779cb3c5992b WatchSource:0}: Error finding container 66bb76393e96fb0922ea2ad79da92c8a4a814490d53764fb8ff1779cb3c5992b: Status 404 returned error can't find the container with id 66bb76393e96fb0922ea2ad79da92c8a4a814490d53764fb8ff1779cb3c5992b Nov 02 22:15:01.268220 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:01.268187 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d94ac34_a86a_4753_b434_606ec862fc3d.slice/crio-09e3de931cbdea09295f8c885269355d8b9bd9a8ec880914747d0c1b0ee3b2dd WatchSource:0}: Error finding container 09e3de931cbdea09295f8c885269355d8b9bd9a8ec880914747d0c1b0ee3b2dd: Status 404 returned error can't find the container with id 09e3de931cbdea09295f8c885269355d8b9bd9a8ec880914747d0c1b0ee3b2dd Nov 02 22:15:01.269796 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:01.269774 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46fe00_01c6_4c24_aa59_0156502f45f2.slice/crio-ec2ba14a52af665fd64b4d9e15dff9ebb9e00d95ec4fddf740a2e343872f5788 WatchSource:0}: Error finding container ec2ba14a52af665fd64b4d9e15dff9ebb9e00d95ec4fddf740a2e343872f5788: Status 404 returned error can't find the container with id ec2ba14a52af665fd64b4d9e15dff9ebb9e00d95ec4fddf740a2e343872f5788 Nov 02 22:15:01.270869 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:01.270843 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810cc234_d11c_4aa8_8754_9e7e3f36d571.slice/crio-207592b801a269b1657261ee1e997c63548bc669faa87a7014f729ecc9d22b36 WatchSource:0}: Error finding container 207592b801a269b1657261ee1e997c63548bc669faa87a7014f729ecc9d22b36: Status 404 returned error can't find the container with id 207592b801a269b1657261ee1e997c63548bc669faa87a7014f729ecc9d22b36 Nov 02 22:15:01.272026 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:01.272003 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad892a2_1fad_4ce6_869c_29b464409490.slice/crio-f3fce4098057936719ab8007e33cbde8fcd9efb72a45ce6b669cf9f88b6041d3 WatchSource:0}: Error finding container f3fce4098057936719ab8007e33cbde8fcd9efb72a45ce6b669cf9f88b6041d3: Status 404 returned error can't find the container with id f3fce4098057936719ab8007e33cbde8fcd9efb72a45ce6b669cf9f88b6041d3 Nov 02 22:15:01.277170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.277144 5091 scope.go:122] "RemoveContainer" containerID="7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825" Nov 02 22:15:01.278014 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:01.277582 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825\": container with ID starting with 7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825 not found: ID does not exist" containerID="7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825" Nov 02 22:15:01.278014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.277620 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825"} err="failed to get container status \"7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825\": rpc error: code = NotFound desc = could not find container \"7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825\": container with ID starting with 7356fe622e2a65d9432e2aec661b3ee7ac10ae162043c4fc72f3188c77695825 not found: ID does not exist" Nov 02 22:15:01.278377 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.278270 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 22:15:01.319701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.319447 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjk5\" (UniqueName: \"kubernetes.io/projected/acb8ba51-d617-41ba-8c4c-9d939c3ce13e-kube-api-access-djjk5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:01.420683 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.420638 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x\" (UniqueName: \"kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x\") pod \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " Nov 02 22:15:01.420936 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.420917 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume\") pod \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\" (UID: \"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf\") " Nov 02 22:15:01.421369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.421340 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume" pod "dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" (UID: "dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:01.422902 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.422874 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x" pod "dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" (UID: "dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf"). InnerVolumeSpecName "kube-api-access-rxk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:01.522370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.522338 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:01.522370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.522366 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxk5x\" (UniqueName: \"kubernetes.io/projected/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf-kube-api-access-rxk5x\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:01.624287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.624258 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn"] Nov 02 22:15:01.627561 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:01.627537 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ffe386-84cb4f87fc-m87gn"] Nov 02 22:15:02.201654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.201621 5091 generic.go:356] "Generic (PLEG): container finished" podID="3c800d43-835a-4235-8bac-196c3eec8b94" containerID="2a4b83c71baaf541ddbf27715010666e072bb18665d8d94cc087bbe4cebf6d1e" exitCode=60 Nov 02 22:15:02.202158 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.201703 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" event={"ID":"3c800d43-835a-4235-8bac-196c3eec8b94","Type":"ContainerDied","Data":"2a4b83c71baaf541ddbf27715010666e072bb18665d8d94cc087bbe4cebf6d1e"} Nov 02 22:15:02.202158 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.201768 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" event={"ID":"3c800d43-835a-4235-8bac-196c3eec8b94","Type":"ContainerStarted","Data":"66bb76393e96fb0922ea2ad79da92c8a4a814490d53764fb8ff1779cb3c5992b"} Nov 02 22:15:02.202158 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.202121 5091 scope.go:122] "RemoveContainer" containerID="2a4b83c71baaf541ddbf27715010666e072bb18665d8d94cc087bbe4cebf6d1e" Nov 02 22:15:02.203972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.203949 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a5af250-15a5-436a-ac4c-66516346482f" containerID="aa2c24a882b30a95c7d761662cf134d38754e5df5654e03287b658ef420d2f3a" exitCode=60 Nov 02 22:15:02.204402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.204383 5091 scope.go:122] "RemoveContainer" containerID="aa2c24a882b30a95c7d761662cf134d38754e5df5654e03287b658ef420d2f3a" Nov 02 22:15:02.204506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.204424 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerDied","Data":"aa2c24a882b30a95c7d761662cf134d38754e5df5654e03287b658ef420d2f3a"} Nov 02 22:15:02.208871 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.207238 5091 generic.go:356] "Generic (PLEG): container finished" podID="d412b8e7-d825-40e3-9ccf-658184e53973" containerID="d0356565aea98204dd03ddc5bc1180a5270e7bc68cf1dedf3ae33e1ba3d376d9" exitCode=60 Nov 02 22:15:02.208871 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.207310 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerDied","Data":"d0356565aea98204dd03ddc5bc1180a5270e7bc68cf1dedf3ae33e1ba3d376d9"} Nov 02 22:15:02.208871 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.207543 5091 scope.go:122] "RemoveContainer" containerID="d0356565aea98204dd03ddc5bc1180a5270e7bc68cf1dedf3ae33e1ba3d376d9" Nov 02 22:15:02.209379 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.209358 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerStarted","Data":"6ce00bb5b31cea373d028022b4556c3db77814aed975bb87561e6a1ba2f08644"} Nov 02 22:15:02.209479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.209401 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerStarted","Data":"f3fce4098057936719ab8007e33cbde8fcd9efb72a45ce6b669cf9f88b6041d3"} Nov 02 22:15:02.211137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.211111 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerStarted","Data":"25c96bd79c9f03d73c7c496aa01c34286f8987c06683043020ef195f2710b5b5"} Nov 02 22:15:02.211237 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.211155 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerStarted","Data":"207592b801a269b1657261ee1e997c63548bc669faa87a7014f729ecc9d22b36"} Nov 02 22:15:02.213397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.213375 5091 generic.go:356] "Generic (PLEG): container finished" podID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerID="7dd2870f0c7b3101241283b97b8317c9b094889d7c94660165b96cf6fbd89209" exitCode=60 Nov 02 22:15:02.213519 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.213452 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerDied","Data":"7dd2870f0c7b3101241283b97b8317c9b094889d7c94660165b96cf6fbd89209"} Nov 02 22:15:02.213650 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.213633 5091 scope.go:122] "RemoveContainer" containerID="7dd2870f0c7b3101241283b97b8317c9b094889d7c94660165b96cf6fbd89209" Nov 02 22:15:02.217275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.217196 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerStarted","Data":"f8ee9817330a698d4fb759da5d1defdb4a8b91b1638a9af7154b49653bf3b17d"} Nov 02 22:15:02.217275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.217238 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerStarted","Data":"09e3de931cbdea09295f8c885269355d8b9bd9a8ec880914747d0c1b0ee3b2dd"} Nov 02 22:15:02.226861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.226826 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" event={"ID":"fe46fe00-01c6-4c24-aa59-0156502f45f2","Type":"ContainerStarted","Data":"2b608da068b2f59ad5683eecb6e76b85ba8231ffc852f4bb6aae602929393a32"} Nov 02 22:15:02.226963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.226888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" event={"ID":"fe46fe00-01c6-4c24-aa59-0156502f45f2","Type":"ContainerStarted","Data":"ec2ba14a52af665fd64b4d9e15dff9ebb9e00d95ec4fddf740a2e343872f5788"} Nov 02 22:15:02.235404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.232341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" event={"ID":"dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf","Type":"ContainerDied","Data":"892d652dd7dd6670de1ef93ac886252518dc7c0041fa8c9b1b6250a048082862"} Nov 02 22:15:02.235404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.232395 5091 scope.go:122] "RemoveContainer" containerID="baf3ad22e0f711b04be34b0d490943a9b8a3f15e79f4bccd38bc8da02ce15de0" Nov 02 22:15:02.235404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.232470 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5" Nov 02 22:15:02.480942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.480881 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" podStartSLOduration=2.480861379 podStartE2EDuration="2.480861379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.478303718 +0000 UTC m=+2449.640398837" watchObservedRunningTime="2026-11-02 22:15:02.480861379 +0000 UTC m=+2449.642956500" Nov 02 22:15:02.594436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.594382 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" podStartSLOduration=2.594363436 podStartE2EDuration="2.594363436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.589269707 +0000 UTC m=+2449.751364829" watchObservedRunningTime="2026-11-02 22:15:02.594363436 +0000 UTC m=+2449.756458558" Nov 02 22:15:02.814320 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.814266 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" podStartSLOduration=2.814251299 podStartE2EDuration="2.814251299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.811534346 +0000 UTC m=+2449.973629467" watchObservedRunningTime="2026-11-02 22:15:02.814251299 +0000 UTC m=+2449.976346418" Nov 02 22:15:02.981609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:02.981581 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" path="/var/lib/kubelet/pods/acb8ba51-d617-41ba-8c4c-9d939c3ce13e/volumes" Nov 02 22:15:03.147310 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.147210 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" podStartSLOduration=3.147197602 podStartE2EDuration="3.147197602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.1452706 +0000 UTC m=+2450.307365723" watchObservedRunningTime="2026-11-02 22:15:03.147197602 +0000 UTC m=+2450.309292722" Nov 02 22:15:03.243619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.243584 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" event={"ID":"3c800d43-835a-4235-8bac-196c3eec8b94","Type":"ContainerStarted","Data":"52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5"} Nov 02 22:15:03.245649 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.245618 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerStarted","Data":"264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1"} Nov 02 22:15:03.247743 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.247707 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerStarted","Data":"688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af"} Nov 02 22:15:03.249314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.249290 5091 generic.go:356] "Generic (PLEG): container finished" podID="5ad892a2-1fad-4ce6-869c-29b464409490" containerID="6ce00bb5b31cea373d028022b4556c3db77814aed975bb87561e6a1ba2f08644" exitCode=60 Nov 02 22:15:03.249417 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.249337 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerDied","Data":"6ce00bb5b31cea373d028022b4556c3db77814aed975bb87561e6a1ba2f08644"} Nov 02 22:15:03.249574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.249556 5091 scope.go:122] "RemoveContainer" containerID="6ce00bb5b31cea373d028022b4556c3db77814aed975bb87561e6a1ba2f08644" Nov 02 22:15:03.251228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.251206 5091 generic.go:356] "Generic (PLEG): container finished" podID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerID="25c96bd79c9f03d73c7c496aa01c34286f8987c06683043020ef195f2710b5b5" exitCode=60 Nov 02 22:15:03.251316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.251233 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerDied","Data":"25c96bd79c9f03d73c7c496aa01c34286f8987c06683043020ef195f2710b5b5"} Nov 02 22:15:03.251535 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.251510 5091 scope.go:122] "RemoveContainer" containerID="25c96bd79c9f03d73c7c496aa01c34286f8987c06683043020ef195f2710b5b5" Nov 02 22:15:03.255003 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.254977 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerStarted","Data":"91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620"} Nov 02 22:15:03.274694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.271357 5091 generic.go:356] "Generic (PLEG): container finished" podID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerID="f8ee9817330a698d4fb759da5d1defdb4a8b91b1638a9af7154b49653bf3b17d" exitCode=60 Nov 02 22:15:03.274694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.271493 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerDied","Data":"f8ee9817330a698d4fb759da5d1defdb4a8b91b1638a9af7154b49653bf3b17d"} Nov 02 22:15:03.274694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.273654 5091 generic.go:356] "Generic (PLEG): container finished" podID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerID="2b608da068b2f59ad5683eecb6e76b85ba8231ffc852f4bb6aae602929393a32" exitCode=60 Nov 02 22:15:03.274929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.274886 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5"] Nov 02 22:15:03.274991 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.274940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" event={"ID":"fe46fe00-01c6-4c24-aa59-0156502f45f2","Type":"ContainerDied","Data":"2b608da068b2f59ad5683eecb6e76b85ba8231ffc852f4bb6aae602929393a32"} Nov 02 22:15:03.278972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.278950 5091 scope.go:122] "RemoveContainer" containerID="f8ee9817330a698d4fb759da5d1defdb4a8b91b1638a9af7154b49653bf3b17d" Nov 02 22:15:03.279100 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.279084 5091 scope.go:122] "RemoveContainer" containerID="2b608da068b2f59ad5683eecb6e76b85ba8231ffc852f4bb6aae602929393a32" Nov 02 22:15:03.279651 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.279582 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ffe386-544c7c7955-ljhg5"] Nov 02 22:15:03.656493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.648719 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" podStartSLOduration=3.648699207 podStartE2EDuration="3.648699207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.512330287 +0000 UTC m=+2450.674425408" watchObservedRunningTime="2026-11-02 22:15:03.648699207 +0000 UTC m=+2450.810794328" Nov 02 22:15:03.808752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.807593 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" podStartSLOduration=3.807572877 podStartE2EDuration="3.807572877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.799776518 +0000 UTC m=+2450.961871639" watchObservedRunningTime="2026-11-02 22:15:03.807572877 +0000 UTC m=+2450.969667998" Nov 02 22:15:03.917889 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:03.917823 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" podStartSLOduration=3.917801752 podStartE2EDuration="3.917801752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.913701958 +0000 UTC m=+2451.075797079" watchObservedRunningTime="2026-11-02 22:15:03.917801752 +0000 UTC m=+2451.079896873" Nov 02 22:15:04.053749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.051824 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" podStartSLOduration=4.051804727 podStartE2EDuration="4.051804727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:04.048360302 +0000 UTC m=+2451.210455424" watchObservedRunningTime="2026-11-02 22:15:04.051804727 +0000 UTC m=+2451.213899850" Nov 02 22:15:04.287294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.287259 5091 generic.go:356] "Generic (PLEG): container finished" podID="3c800d43-835a-4235-8bac-196c3eec8b94" containerID="52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5" exitCode=60 Nov 02 22:15:04.287806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.287349 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" event={"ID":"3c800d43-835a-4235-8bac-196c3eec8b94","Type":"ContainerDied","Data":"52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5"} Nov 02 22:15:04.287806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.287421 5091 scope.go:122] "RemoveContainer" containerID="2a4b83c71baaf541ddbf27715010666e072bb18665d8d94cc087bbe4cebf6d1e" Nov 02 22:15:04.287806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.287737 5091 scope.go:122] "RemoveContainer" containerID="52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5" Nov 02 22:15:04.288094 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:04.288065 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh_odh-ai-gateway-infra(3c800d43-835a-4235-8bac-196c3eec8b94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" Nov 02 22:15:04.289974 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.289950 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a5af250-15a5-436a-ac4c-66516346482f" containerID="264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1" exitCode=60 Nov 02 22:15:04.290091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.290050 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerDied","Data":"264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1"} Nov 02 22:15:04.290304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.290286 5091 scope.go:122] "RemoveContainer" containerID="264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1" Nov 02 22:15:04.290589 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:04.290565 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv_odh-ai-gateway-infra(0a5af250-15a5-436a-ac4c-66516346482f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" podUID="0a5af250-15a5-436a-ac4c-66516346482f" Nov 02 22:15:04.292203 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.292184 5091 generic.go:356] "Generic (PLEG): container finished" podID="d412b8e7-d825-40e3-9ccf-658184e53973" containerID="688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af" exitCode=60 Nov 02 22:15:04.292307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.292245 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerDied","Data":"688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af"} Nov 02 22:15:04.292566 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.292546 5091 scope.go:122] "RemoveContainer" containerID="688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af" Nov 02 22:15:04.292905 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:04.292881 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm_odh-ai-gateway-infra(d412b8e7-d825-40e3-9ccf-658184e53973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" Nov 02 22:15:04.294544 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.294521 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerStarted","Data":"ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073"} Nov 02 22:15:04.297684 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.297650 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerStarted","Data":"28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf"} Nov 02 22:15:04.302907 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.302885 5091 generic.go:356] "Generic (PLEG): container finished" podID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerID="91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620" exitCode=60 Nov 02 22:15:04.303015 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.302957 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerDied","Data":"91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620"} Nov 02 22:15:04.303200 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.303186 5091 scope.go:122] "RemoveContainer" containerID="91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620" Nov 02 22:15:04.303422 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:04.303403 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2_odh-ai-gateway-infra(6944953f-a3de-4004-a4f3-9f748d66d1c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" Nov 02 22:15:04.305388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.305358 5091 scope.go:122] "RemoveContainer" containerID="aa2c24a882b30a95c7d761662cf134d38754e5df5654e03287b658ef420d2f3a" Nov 02 22:15:04.305924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.305894 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerStarted","Data":"b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b"} Nov 02 22:15:04.308125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.308101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" event={"ID":"fe46fe00-01c6-4c24-aa59-0156502f45f2","Type":"ContainerStarted","Data":"116d0fc900fe53f258aeb33ecf720ee36fac4383fe61a83d3108d79d4b45b197"} Nov 02 22:15:04.321118 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.321094 5091 scope.go:122] "RemoveContainer" containerID="d0356565aea98204dd03ddc5bc1180a5270e7bc68cf1dedf3ae33e1ba3d376d9" Nov 02 22:15:04.336744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.336721 5091 scope.go:122] "RemoveContainer" containerID="7dd2870f0c7b3101241283b97b8317c9b094889d7c94660165b96cf6fbd89209" Nov 02 22:15:04.988869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:04.988826 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" path="/var/lib/kubelet/pods/dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf/volumes" Nov 02 22:15:05.318090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.317998 5091 scope.go:122] "RemoveContainer" containerID="52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5" Nov 02 22:15:05.318588 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.318344 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh_odh-ai-gateway-infra(3c800d43-835a-4235-8bac-196c3eec8b94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" Nov 02 22:15:05.319920 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.319894 5091 scope.go:122] "RemoveContainer" containerID="264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1" Nov 02 22:15:05.320180 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.320156 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv_odh-ai-gateway-infra(0a5af250-15a5-436a-ac4c-66516346482f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" podUID="0a5af250-15a5-436a-ac4c-66516346482f" Nov 02 22:15:05.321758 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.321727 5091 scope.go:122] "RemoveContainer" containerID="688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af" Nov 02 22:15:05.322008 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.321985 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm_odh-ai-gateway-infra(d412b8e7-d825-40e3-9ccf-658184e53973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" Nov 02 22:15:05.323615 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.323595 5091 generic.go:356] "Generic (PLEG): container finished" podID="5ad892a2-1fad-4ce6-869c-29b464409490" containerID="ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073" exitCode=60 Nov 02 22:15:05.323768 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.323735 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerDied","Data":"ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073"} Nov 02 22:15:05.323856 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.323819 5091 scope.go:122] "RemoveContainer" containerID="6ce00bb5b31cea373d028022b4556c3db77814aed975bb87561e6a1ba2f08644" Nov 02 22:15:05.323917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.323884 5091 scope.go:122] "RemoveContainer" containerID="ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073" Nov 02 22:15:05.324142 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.324119 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz_odh-ai-gateway-infra(5ad892a2-1fad-4ce6-869c-29b464409490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" Nov 02 22:15:05.325927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.325904 5091 generic.go:356] "Generic (PLEG): container finished" podID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerID="28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf" exitCode=60 Nov 02 22:15:05.326025 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.325985 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerDied","Data":"28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf"} Nov 02 22:15:05.326162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.326146 5091 scope.go:122] "RemoveContainer" containerID="28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf" Nov 02 22:15:05.326405 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.326383 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg_odh-ai-gateway-infra(810cc234-d11c-4aa8-8754-9e7e3f36d571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" Nov 02 22:15:05.327816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.327799 5091 scope.go:122] "RemoveContainer" containerID="91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620" Nov 02 22:15:05.327971 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.327952 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2_odh-ai-gateway-infra(6944953f-a3de-4004-a4f3-9f748d66d1c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" Nov 02 22:15:05.329833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.329809 5091 generic.go:356] "Generic (PLEG): container finished" podID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerID="b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b" exitCode=60 Nov 02 22:15:05.329929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.329897 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerDied","Data":"b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b"} Nov 02 22:15:05.330088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.330072 5091 scope.go:122] "RemoveContainer" containerID="b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b" Nov 02 22:15:05.330262 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.330244 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2_odh-ai-gateway-infra(2d94ac34-a86a-4753-b434-606ec862fc3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" Nov 02 22:15:05.331942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.331924 5091 generic.go:356] "Generic (PLEG): container finished" podID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerID="116d0fc900fe53f258aeb33ecf720ee36fac4383fe61a83d3108d79d4b45b197" exitCode=60 Nov 02 22:15:05.332023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.331982 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" event={"ID":"fe46fe00-01c6-4c24-aa59-0156502f45f2","Type":"ContainerDied","Data":"116d0fc900fe53f258aeb33ecf720ee36fac4383fe61a83d3108d79d4b45b197"} Nov 02 22:15:05.332174 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.332158 5091 scope.go:122] "RemoveContainer" containerID="116d0fc900fe53f258aeb33ecf720ee36fac4383fe61a83d3108d79d4b45b197" Nov 02 22:15:05.332334 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:05.332320 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k_odh-ai-gateway-infra(fe46fe00-01c6-4c24-aa59-0156502f45f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" Nov 02 22:15:05.337420 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.337403 5091 scope.go:122] "RemoveContainer" containerID="25c96bd79c9f03d73c7c496aa01c34286f8987c06683043020ef195f2710b5b5" Nov 02 22:15:05.352094 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.352074 5091 scope.go:122] "RemoveContainer" containerID="f8ee9817330a698d4fb759da5d1defdb4a8b91b1638a9af7154b49653bf3b17d" Nov 02 22:15:05.366335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:05.366315 5091 scope.go:122] "RemoveContainer" containerID="2b608da068b2f59ad5683eecb6e76b85ba8231ffc852f4bb6aae602929393a32" Nov 02 22:15:06.343955 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:06.343926 5091 scope.go:122] "RemoveContainer" containerID="ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073" Nov 02 22:15:06.344457 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:06.344234 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz_odh-ai-gateway-infra(5ad892a2-1fad-4ce6-869c-29b464409490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" Nov 02 22:15:06.350888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:06.350865 5091 scope.go:122] "RemoveContainer" containerID="28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf" Nov 02 22:15:06.351128 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:06.351103 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg_odh-ai-gateway-infra(810cc234-d11c-4aa8-8754-9e7e3f36d571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" Nov 02 22:15:06.353056 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:06.353035 5091 scope.go:122] "RemoveContainer" containerID="b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b" Nov 02 22:15:06.353245 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:06.353227 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2_odh-ai-gateway-infra(2d94ac34-a86a-4753-b434-606ec862fc3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" Nov 02 22:15:06.355932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:06.355911 5091 scope.go:122] "RemoveContainer" containerID="116d0fc900fe53f258aeb33ecf720ee36fac4383fe61a83d3108d79d4b45b197" Nov 02 22:15:06.356160 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:06.356139 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k_odh-ai-gateway-infra(fe46fe00-01c6-4c24-aa59-0156502f45f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" Nov 02 22:15:07.113132 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:07.113100 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:15:14.971526 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.971491 5091 scope.go:122] "RemoveContainer" containerID="91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620" Nov 02 22:15:14.972060 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.971732 5091 scope.go:122] "RemoveContainer" containerID="52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5" Nov 02 22:15:14.979392 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.979370 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw"] Nov 02 22:15:14.979908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.979894 5091 cpu_manager.go:414] "containerMap: removing container" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerName="payload-processing" Nov 02 22:15:14.979953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.979909 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerName="payload-processing" Nov 02 22:15:14.979953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.979944 5091 cpu_manager.go:414] "containerMap: removing container" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerName="payload-pre-processing" Nov 02 22:15:14.979953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.979948 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerName="payload-pre-processing" Nov 02 22:15:14.980053 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.980042 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd54ecb3-9693-4cf3-b36c-4ce7483bf5cf" containerName="payload-pre-processing" Nov 02 22:15:14.980085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.980054 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb8ba51-d617-41ba-8c4c-9d939c3ce13e" containerName="payload-processing" Nov 02 22:15:14.984536 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:14.984519 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.019837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.019810 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmlk\" (UniqueName: \"kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.020215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.020186 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.020377 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.020359 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.037702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.037665 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw"] Nov 02 22:15:15.082126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.082093 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:15.122129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.122091 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.122300 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.122211 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmlk\" (UniqueName: \"kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.122363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.122315 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.123318 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.123270 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.163614 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.163582 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.178790 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.178763 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmlk\" (UniqueName: \"kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk\") pod \"maas-api-revoke-sub-af7ab6b5-6glcw\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.267519 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.267443 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k"] Nov 02 22:15:15.346328 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.346289 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:15.433481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.432612 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerStarted","Data":"b0b99c8dffb0f10cf719f21f2c030cc623b8c730f271cfc35645db41cd3e74fe"} Nov 02 22:15:15.482285 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.482251 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:15.655696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.653269 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h\" (UniqueName: \"kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h\") pod \"fe46fe00-01c6-4c24-aa59-0156502f45f2\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " Nov 02 22:15:15.655696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.653439 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca\") pod \"fe46fe00-01c6-4c24-aa59-0156502f45f2\" (UID: \"fe46fe00-01c6-4c24-aa59-0156502f45f2\") " Nov 02 22:15:15.657323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.656097 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca" pod "fe46fe00-01c6-4c24-aa59-0156502f45f2" (UID: "fe46fe00-01c6-4c24-aa59-0156502f45f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:15.667014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.659606 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h" pod "fe46fe00-01c6-4c24-aa59-0156502f45f2" (UID: "fe46fe00-01c6-4c24-aa59-0156502f45f2"). InnerVolumeSpecName "kube-api-access-6sk6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:15.754950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.754915 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sk6h\" (UniqueName: \"kubernetes.io/projected/fe46fe00-01c6-4c24-aa59-0156502f45f2-kube-api-access-6sk6h\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:15.754950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.754943 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe46fe00-01c6-4c24-aa59-0156502f45f2-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:15.797709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.797661 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw"] Nov 02 22:15:15.798864 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:15.798838 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ca119_617d_4312_9f91_59faea5cc71f.slice/crio-4c34c6fa5d73c365f7c81d4e63394478df0a6a62f6b64d5c4abb605929242ecb WatchSource:0}: Error finding container 4c34c6fa5d73c365f7c81d4e63394478df0a6a62f6b64d5c4abb605929242ecb: Status 404 returned error can't find the container with id 4c34c6fa5d73c365f7c81d4e63394478df0a6a62f6b64d5c4abb605929242ecb Nov 02 22:15:15.971494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:15.971025 5091 scope.go:122] "RemoveContainer" containerID="28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf" Nov 02 22:15:16.450214 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.450179 5091 generic.go:356] "Generic (PLEG): container finished" podID="3c800d43-835a-4235-8bac-196c3eec8b94" containerID="9254c7b2ae72574ba4bec20481503bf9d47e9983296408b9e090ad0a98ac3640" exitCode=60 Nov 02 22:15:16.450606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.450251 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" event={"ID":"3c800d43-835a-4235-8bac-196c3eec8b94","Type":"ContainerDied","Data":"9254c7b2ae72574ba4bec20481503bf9d47e9983296408b9e090ad0a98ac3640"} Nov 02 22:15:16.450606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.450338 5091 scope.go:122] "RemoveContainer" containerID="52cdfe4cc2c6fec24c913956c42469ba90100a426b39d6553869075afbc1b6d5" Nov 02 22:15:16.450751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.450731 5091 scope.go:122] "RemoveContainer" containerID="9254c7b2ae72574ba4bec20481503bf9d47e9983296408b9e090ad0a98ac3640" Nov 02 22:15:16.451051 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:16.451022 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh_odh-ai-gateway-infra(3c800d43-835a-4235-8bac-196c3eec8b94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" Nov 02 22:15:16.452122 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.452097 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" event={"ID":"ab0ca119-617d-4312-9f91-59faea5cc71f","Type":"ContainerStarted","Data":"f237500afb3bdca6c836adbd8d50d8a57be9b72aca82d94595e507701f90cac4"} Nov 02 22:15:16.452225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.452145 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" event={"ID":"ab0ca119-617d-4312-9f91-59faea5cc71f","Type":"ContainerStarted","Data":"4c34c6fa5d73c365f7c81d4e63394478df0a6a62f6b64d5c4abb605929242ecb"} Nov 02 22:15:16.453943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.453917 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerStarted","Data":"ef476ce8e32623818872ccd11327f054de169c925241142984b74278d7756ccc"} Nov 02 22:15:16.455842 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.455808 5091 generic.go:356] "Generic (PLEG): container finished" podID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerID="b0b99c8dffb0f10cf719f21f2c030cc623b8c730f271cfc35645db41cd3e74fe" exitCode=60 Nov 02 22:15:16.455956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.455877 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerDied","Data":"b0b99c8dffb0f10cf719f21f2c030cc623b8c730f271cfc35645db41cd3e74fe"} Nov 02 22:15:16.456147 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.456129 5091 scope.go:122] "RemoveContainer" containerID="b0b99c8dffb0f10cf719f21f2c030cc623b8c730f271cfc35645db41cd3e74fe" Nov 02 22:15:16.456373 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:16.456350 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2_odh-ai-gateway-infra(6944953f-a3de-4004-a4f3-9f748d66d1c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" Nov 02 22:15:16.457232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.457213 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" Nov 02 22:15:16.457316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.457234 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k" event={"ID":"fe46fe00-01c6-4c24-aa59-0156502f45f2","Type":"ContainerDied","Data":"ec2ba14a52af665fd64b4d9e15dff9ebb9e00d95ec4fddf740a2e343872f5788"} Nov 02 22:15:16.464343 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.464327 5091 scope.go:122] "RemoveContainer" containerID="91478522a1229bc784bbd44dc6d9361e36596d090bfea78086e343e0d6782620" Nov 02 22:15:16.477722 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.477701 5091 scope.go:122] "RemoveContainer" containerID="116d0fc900fe53f258aeb33ecf720ee36fac4383fe61a83d3108d79d4b45b197" Nov 02 22:15:16.556467 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.556438 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2"] Nov 02 22:15:16.676757 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.676653 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k"] Nov 02 22:15:16.680252 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.680226 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-64438f-29894295-v677k"] Nov 02 22:15:16.788172 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.788128 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" podStartSLOduration=2.788114437 podStartE2EDuration="2.788114437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:16.785909457 +0000 UTC m=+2463.948004577" watchObservedRunningTime="2026-11-02 22:15:16.788114437 +0000 UTC m=+2463.950209572" Nov 02 22:15:16.969998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.969929 5091 scope.go:122] "RemoveContainer" containerID="688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af" Nov 02 22:15:16.978049 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:16.978019 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" path="/var/lib/kubelet/pods/fe46fe00-01c6-4c24-aa59-0156502f45f2/volumes" Nov 02 22:15:17.368182 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.368147 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp"] Nov 02 22:15:17.371564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.371535 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-64438f-dfgsp"] Nov 02 22:15:17.468357 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.468326 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerStarted","Data":"68333db5308fb545bef13c79a7fb9a20bcc218d861f39b86fd41dba3fece7b5e"} Nov 02 22:15:17.470000 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.469973 5091 generic.go:356] "Generic (PLEG): container finished" podID="ab0ca119-617d-4312-9f91-59faea5cc71f" containerID="f237500afb3bdca6c836adbd8d50d8a57be9b72aca82d94595e507701f90cac4" exitCode=0 Nov 02 22:15:17.470136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.470049 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" event={"ID":"ab0ca119-617d-4312-9f91-59faea5cc71f","Type":"ContainerDied","Data":"f237500afb3bdca6c836adbd8d50d8a57be9b72aca82d94595e507701f90cac4"} Nov 02 22:15:17.471859 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.471843 5091 generic.go:356] "Generic (PLEG): container finished" podID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerID="ef476ce8e32623818872ccd11327f054de169c925241142984b74278d7756ccc" exitCode=60 Nov 02 22:15:17.471942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.471913 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerDied","Data":"ef476ce8e32623818872ccd11327f054de169c925241142984b74278d7756ccc"} Nov 02 22:15:17.471992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.471974 5091 scope.go:122] "RemoveContainer" containerID="28130b8c1c26e3296a4690c8bfe00c83d66b1f975392187cb9465b2b0e181dbf" Nov 02 22:15:17.472258 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.472242 5091 scope.go:122] "RemoveContainer" containerID="ef476ce8e32623818872ccd11327f054de169c925241142984b74278d7756ccc" Nov 02 22:15:17.472506 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:17.472482 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg_odh-ai-gateway-infra(810cc234-d11c-4aa8-8754-9e7e3f36d571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" Nov 02 22:15:17.578585 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.578551 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh"] Nov 02 22:15:17.633340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.633315 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:17.674271 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.674241 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8\" (UniqueName: \"kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8\") pod \"6944953f-a3de-4004-a4f3-9f748d66d1c5\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " Nov 02 22:15:17.676849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.676817 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8" pod "6944953f-a3de-4004-a4f3-9f748d66d1c5" (UID: "6944953f-a3de-4004-a4f3-9f748d66d1c5"). InnerVolumeSpecName "kube-api-access-9mkq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:17.743652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.743628 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:17.775021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.774994 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca\") pod \"6944953f-a3de-4004-a4f3-9f748d66d1c5\" (UID: \"6944953f-a3de-4004-a4f3-9f748d66d1c5\") " Nov 02 22:15:17.775303 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.775283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca" pod "6944953f-a3de-4004-a4f3-9f748d66d1c5" (UID: "6944953f-a3de-4004-a4f3-9f748d66d1c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:17.775372 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.775359 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6944953f-a3de-4004-a4f3-9f748d66d1c5-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.775406 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.775375 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mkq8\" (UniqueName: \"kubernetes.io/projected/6944953f-a3de-4004-a4f3-9f748d66d1c5-kube-api-access-9mkq8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.875836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.875808 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca\") pod \"3c800d43-835a-4235-8bac-196c3eec8b94\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " Nov 02 22:15:17.876002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.875846 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv\" (UniqueName: \"kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv\") pod \"3c800d43-835a-4235-8bac-196c3eec8b94\" (UID: \"3c800d43-835a-4235-8bac-196c3eec8b94\") " Nov 02 22:15:17.876254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.876230 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca" pod "3c800d43-835a-4235-8bac-196c3eec8b94" (UID: "3c800d43-835a-4235-8bac-196c3eec8b94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:17.877954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.877934 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv" pod "3c800d43-835a-4235-8bac-196c3eec8b94" (UID: "3c800d43-835a-4235-8bac-196c3eec8b94"). InnerVolumeSpecName "kube-api-access-wdxvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:17.909005 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.908973 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg"] Nov 02 22:15:17.970203 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.970177 5091 scope.go:122] "RemoveContainer" containerID="ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073" Nov 02 22:15:17.970464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.970441 5091 scope.go:122] "RemoveContainer" containerID="264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1" Nov 02 22:15:17.977056 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.976973 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c800d43-835a-4235-8bac-196c3eec8b94-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.977056 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:17.977000 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxvv\" (UniqueName: \"kubernetes.io/projected/3c800d43-835a-4235-8bac-196c3eec8b94-kube-api-access-wdxvv\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.485068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.484971 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" event={"ID":"3c800d43-835a-4235-8bac-196c3eec8b94","Type":"ContainerDied","Data":"66bb76393e96fb0922ea2ad79da92c8a4a814490d53764fb8ff1779cb3c5992b"} Nov 02 22:15:18.485068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.484991 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh" Nov 02 22:15:18.485604 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.485068 5091 scope.go:122] "RemoveContainer" containerID="9254c7b2ae72574ba4bec20481503bf9d47e9983296408b9e090ad0a98ac3640" Nov 02 22:15:18.487289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.487261 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerStarted","Data":"3e6e9a2abf82b7696f1e3a1e8d8dae0e06fa5323f9a9ddd525db7e2d1807872e"} Nov 02 22:15:18.489277 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.489250 5091 generic.go:356] "Generic (PLEG): container finished" podID="d412b8e7-d825-40e3-9ccf-658184e53973" containerID="68333db5308fb545bef13c79a7fb9a20bcc218d861f39b86fd41dba3fece7b5e" exitCode=60 Nov 02 22:15:18.489389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.489313 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerDied","Data":"68333db5308fb545bef13c79a7fb9a20bcc218d861f39b86fd41dba3fece7b5e"} Nov 02 22:15:18.489699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.489663 5091 scope.go:122] "RemoveContainer" containerID="68333db5308fb545bef13c79a7fb9a20bcc218d861f39b86fd41dba3fece7b5e" Nov 02 22:15:18.489950 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:18.489924 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm_odh-ai-gateway-infra(d412b8e7-d825-40e3-9ccf-658184e53973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" Nov 02 22:15:18.491710 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.491686 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerStarted","Data":"018d19ebd6b4e63cf62abd87c5492b69b0f2698020598490f89afd71a1f533f7"} Nov 02 22:15:18.499715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.498736 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" Nov 02 22:15:18.499715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.498762 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2" event={"ID":"6944953f-a3de-4004-a4f3-9f748d66d1c5","Type":"ContainerDied","Data":"4881264c95c8977450c4bfaccd3170a73cb49dd2033385e1d3e6e45d95933f12"} Nov 02 22:15:18.501823 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.501801 5091 scope.go:122] "RemoveContainer" containerID="688514a78d3c7ab9fb28eb19f983acc637e47e96b566c8579a728f21826a59af" Nov 02 22:15:18.531547 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.529170 5091 scope.go:122] "RemoveContainer" containerID="b0b99c8dffb0f10cf719f21f2c030cc623b8c730f271cfc35645db41cd3e74fe" Nov 02 22:15:18.595898 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.595859 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm"] Nov 02 22:15:18.662805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.662778 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:18.683991 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.683969 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7\" (UniqueName: \"kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7\") pod \"810cc234-d11c-4aa8-8754-9e7e3f36d571\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " Nov 02 22:15:18.684085 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.684068 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca\") pod \"810cc234-d11c-4aa8-8754-9e7e3f36d571\" (UID: \"810cc234-d11c-4aa8-8754-9e7e3f36d571\") " Nov 02 22:15:18.684462 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.684435 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca" pod "810cc234-d11c-4aa8-8754-9e7e3f36d571" (UID: "810cc234-d11c-4aa8-8754-9e7e3f36d571"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:18.684598 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.684539 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810cc234-d11c-4aa8-8754-9e7e3f36d571-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.686126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.686097 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7" pod "810cc234-d11c-4aa8-8754-9e7e3f36d571" (UID: "810cc234-d11c-4aa8-8754-9e7e3f36d571"). InnerVolumeSpecName "kube-api-access-r45g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:18.711699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.711645 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:18.785106 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.785074 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca\") pod \"ab0ca119-617d-4312-9f91-59faea5cc71f\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " Nov 02 22:15:18.785266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.785176 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca\") pod \"ab0ca119-617d-4312-9f91-59faea5cc71f\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " Nov 02 22:15:18.785266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.785210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk\" (UniqueName: \"kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk\") pod \"ab0ca119-617d-4312-9f91-59faea5cc71f\" (UID: \"ab0ca119-617d-4312-9f91-59faea5cc71f\") " Nov 02 22:15:18.785573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.785548 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca" pod "ab0ca119-617d-4312-9f91-59faea5cc71f" (UID: "ab0ca119-617d-4312-9f91-59faea5cc71f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:18.785655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.785609 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r45g7\" (UniqueName: \"kubernetes.io/projected/810cc234-d11c-4aa8-8754-9e7e3f36d571-kube-api-access-r45g7\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.787110 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.787090 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca" pod "ab0ca119-617d-4312-9f91-59faea5cc71f" (UID: "ab0ca119-617d-4312-9f91-59faea5cc71f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:18.787295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.787276 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk" pod "ab0ca119-617d-4312-9f91-59faea5cc71f" (UID: "ab0ca119-617d-4312-9f91-59faea5cc71f"). InnerVolumeSpecName "kube-api-access-dgmlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:18.887241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.887209 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.887241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.887239 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ca119-617d-4312-9f91-59faea5cc71f-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.887433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.887249 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmlk\" (UniqueName: \"kubernetes.io/projected/ab0ca119-617d-4312-9f91-59faea5cc71f-kube-api-access-dgmlk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.969883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.969857 5091 scope.go:122] "RemoveContainer" containerID="b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b" Nov 02 22:15:18.977308 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.977277 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c976f4bf-44ce-432d-b554-8db925933215" path="/var/lib/kubelet/pods/c976f4bf-44ce-432d-b554-8db925933215/volumes" Nov 02 22:15:18.990373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.990347 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2"] Nov 02 22:15:18.994356 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:18.994333 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894295-89ms2"] Nov 02 22:15:19.101590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.101554 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh"] Nov 02 22:15:19.104826 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.104798 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894295-hrqdh"] Nov 02 22:15:19.508933 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.508902 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a5af250-15a5-436a-ac4c-66516346482f" containerID="3e6e9a2abf82b7696f1e3a1e8d8dae0e06fa5323f9a9ddd525db7e2d1807872e" exitCode=60 Nov 02 22:15:19.509469 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.508979 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerDied","Data":"3e6e9a2abf82b7696f1e3a1e8d8dae0e06fa5323f9a9ddd525db7e2d1807872e"} Nov 02 22:15:19.509469 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.509114 5091 scope.go:122] "RemoveContainer" containerID="264871012dc5300d62ff6a56b051994ba7e2bcda23cc4d6ec738d38f2d3efed1" Nov 02 22:15:19.509657 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.509548 5091 scope.go:122] "RemoveContainer" containerID="3e6e9a2abf82b7696f1e3a1e8d8dae0e06fa5323f9a9ddd525db7e2d1807872e" Nov 02 22:15:19.509930 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:19.509889 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv_odh-ai-gateway-infra(0a5af250-15a5-436a-ac4c-66516346482f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" podUID="0a5af250-15a5-436a-ac4c-66516346482f" Nov 02 22:15:19.513568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.513546 5091 generic.go:356] "Generic (PLEG): container finished" podID="5ad892a2-1fad-4ce6-869c-29b464409490" containerID="018d19ebd6b4e63cf62abd87c5492b69b0f2698020598490f89afd71a1f533f7" exitCode=60 Nov 02 22:15:19.513695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.513599 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerDied","Data":"018d19ebd6b4e63cf62abd87c5492b69b0f2698020598490f89afd71a1f533f7"} Nov 02 22:15:19.513996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.513974 5091 scope.go:122] "RemoveContainer" containerID="018d19ebd6b4e63cf62abd87c5492b69b0f2698020598490f89afd71a1f533f7" Nov 02 22:15:19.514308 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:19.514282 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz_odh-ai-gateway-infra(5ad892a2-1fad-4ce6-869c-29b464409490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" Nov 02 22:15:19.515364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.515342 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" Nov 02 22:15:19.515479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.515367 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw" event={"ID":"ab0ca119-617d-4312-9f91-59faea5cc71f","Type":"ContainerDied","Data":"4c34c6fa5d73c365f7c81d4e63394478df0a6a62f6b64d5c4abb605929242ecb"} Nov 02 22:15:19.515479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.515409 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c34c6fa5d73c365f7c81d4e63394478df0a6a62f6b64d5c4abb605929242ecb" Nov 02 22:15:19.516885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.516859 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" event={"ID":"810cc234-d11c-4aa8-8754-9e7e3f36d571","Type":"ContainerDied","Data":"207592b801a269b1657261ee1e997c63548bc669faa87a7014f729ecc9d22b36"} Nov 02 22:15:19.517000 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.516865 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg" Nov 02 22:15:19.520314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.520290 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerStarted","Data":"fff5e8ab81ec5d7f1841de4affe890500e8615c8122251dd4374e60b1b89f060"} Nov 02 22:15:19.525433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.525414 5091 scope.go:122] "RemoveContainer" containerID="ba01feb414e0d7ffec5cfb6e63b33c3a1f85190ba11922c72c9b340802890073" Nov 02 22:15:19.540975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.540954 5091 scope.go:122] "RemoveContainer" containerID="ef476ce8e32623818872ccd11327f054de169c925241142984b74278d7756ccc" Nov 02 22:15:19.666243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.666207 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz"] Nov 02 22:15:19.683440 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.683416 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:19.696703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.696660 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca\") pod \"d412b8e7-d825-40e3-9ccf-658184e53973\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " Nov 02 22:15:19.696844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.696753 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr\" (UniqueName: \"kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr\") pod \"d412b8e7-d825-40e3-9ccf-658184e53973\" (UID: \"d412b8e7-d825-40e3-9ccf-658184e53973\") " Nov 02 22:15:19.696970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.696948 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca" pod "d412b8e7-d825-40e3-9ccf-658184e53973" (UID: "d412b8e7-d825-40e3-9ccf-658184e53973"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:19.697243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.697227 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d412b8e7-d825-40e3-9ccf-658184e53973-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.698827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.698808 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr" pod "d412b8e7-d825-40e3-9ccf-658184e53973" (UID: "d412b8e7-d825-40e3-9ccf-658184e53973"). InnerVolumeSpecName "kube-api-access-j4wrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:19.776720 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.776637 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv"] Nov 02 22:15:19.798235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:19.798206 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wrr\" (UniqueName: \"kubernetes.io/projected/d412b8e7-d825-40e3-9ccf-658184e53973-kube-api-access-j4wrr\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.117539 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.117517 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg"] Nov 02 22:15:20.121083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.121054 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d1a155-29894295-llqlg"] Nov 02 22:15:20.448124 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.448046 5091 scope.go:122] "RemoveContainer" containerID="e3f7fdfecc20e9312858813f7b995a92d62c26b6e65dada64023dd549db3f951" Nov 02 22:15:20.532446 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.532397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" event={"ID":"d412b8e7-d825-40e3-9ccf-658184e53973","Type":"ContainerDied","Data":"747001f911648d2777afc1e802970dea9c03d7f919f0d78548061cb6574b5b8d"} Nov 02 22:15:20.532840 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.532515 5091 scope.go:122] "RemoveContainer" containerID="68333db5308fb545bef13c79a7fb9a20bcc218d861f39b86fd41dba3fece7b5e" Nov 02 22:15:20.532840 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.532421 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm" Nov 02 22:15:20.541459 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.541432 5091 generic.go:356] "Generic (PLEG): container finished" podID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerID="fff5e8ab81ec5d7f1841de4affe890500e8615c8122251dd4374e60b1b89f060" exitCode=60 Nov 02 22:15:20.541665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.541640 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerDied","Data":"fff5e8ab81ec5d7f1841de4affe890500e8615c8122251dd4374e60b1b89f060"} Nov 02 22:15:20.542123 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.542088 5091 scope.go:122] "RemoveContainer" containerID="fff5e8ab81ec5d7f1841de4affe890500e8615c8122251dd4374e60b1b89f060" Nov 02 22:15:20.542472 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:20.542447 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2_odh-ai-gateway-infra(2d94ac34-a86a-4753-b434-606ec862fc3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" Nov 02 22:15:20.547963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.547868 5091 scope.go:122] "RemoveContainer" containerID="b1545587fcdb489fa9901acacb342b6980f1535117f6252a416d124d67404c9b" Nov 02 22:15:20.696119 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.696086 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:20.708474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.708403 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca\") pod \"0a5af250-15a5-436a-ac4c-66516346482f\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " Nov 02 22:15:20.708632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.708610 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v\" (UniqueName: \"kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v\") pod \"0a5af250-15a5-436a-ac4c-66516346482f\" (UID: \"0a5af250-15a5-436a-ac4c-66516346482f\") " Nov 02 22:15:20.708796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.708774 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca" pod "0a5af250-15a5-436a-ac4c-66516346482f" (UID: "0a5af250-15a5-436a-ac4c-66516346482f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:20.709140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.709120 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5af250-15a5-436a-ac4c-66516346482f-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.710900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.710874 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v" pod "0a5af250-15a5-436a-ac4c-66516346482f" (UID: "0a5af250-15a5-436a-ac4c-66516346482f"). InnerVolumeSpecName "kube-api-access-llk4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:20.738629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.738611 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:20.756481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.756459 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm"] Nov 02 22:15:20.763550 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.763525 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb8112-29894295-9qksm"] Nov 02 22:15:20.809519 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.809483 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc\" (UniqueName: \"kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc\") pod \"5ad892a2-1fad-4ce6-869c-29b464409490\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " Nov 02 22:15:20.809704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.809568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca\") pod \"5ad892a2-1fad-4ce6-869c-29b464409490\" (UID: \"5ad892a2-1fad-4ce6-869c-29b464409490\") " Nov 02 22:15:20.809989 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.809972 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llk4v\" (UniqueName: \"kubernetes.io/projected/0a5af250-15a5-436a-ac4c-66516346482f-kube-api-access-llk4v\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.810058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.809967 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca" pod "5ad892a2-1fad-4ce6-869c-29b464409490" (UID: "5ad892a2-1fad-4ce6-869c-29b464409490"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:20.811780 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.811755 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc" pod "5ad892a2-1fad-4ce6-869c-29b464409490" (UID: "5ad892a2-1fad-4ce6-869c-29b464409490"). InnerVolumeSpecName "kube-api-access-pp7qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:20.910799 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.910764 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7qc\" (UniqueName: \"kubernetes.io/projected/5ad892a2-1fad-4ce6-869c-29b464409490-kube-api-access-pp7qc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.910799 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.910790 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad892a2-1fad-4ce6-869c-29b464409490-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.977456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.977382 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" path="/var/lib/kubelet/pods/3c800d43-835a-4235-8bac-196c3eec8b94/volumes" Nov 02 22:15:20.977945 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.977929 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" path="/var/lib/kubelet/pods/6944953f-a3de-4004-a4f3-9f748d66d1c5/volumes" Nov 02 22:15:20.978364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.978350 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" path="/var/lib/kubelet/pods/810cc234-d11c-4aa8-8754-9e7e3f36d571/volumes" Nov 02 22:15:20.979111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:20.979094 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" path="/var/lib/kubelet/pods/d412b8e7-d825-40e3-9ccf-658184e53973/volumes" Nov 02 22:15:21.017314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.017285 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2"] Nov 02 22:15:21.549712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.549660 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" Nov 02 22:15:21.549712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.549704 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv" event={"ID":"0a5af250-15a5-436a-ac4c-66516346482f","Type":"ContainerDied","Data":"ce057a6cb8e670c24a97175e33d812ad012ba873df3a04f3ec781e6e811b6830"} Nov 02 22:15:21.550204 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.549770 5091 scope.go:122] "RemoveContainer" containerID="3e6e9a2abf82b7696f1e3a1e8d8dae0e06fa5323f9a9ddd525db7e2d1807872e" Nov 02 22:15:21.552770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.552753 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" Nov 02 22:15:21.552902 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.552789 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz" event={"ID":"5ad892a2-1fad-4ce6-869c-29b464409490","Type":"ContainerDied","Data":"f3fce4098057936719ab8007e33cbde8fcd9efb72a45ce6b669cf9f88b6041d3"} Nov 02 22:15:21.560913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.560891 5091 scope.go:122] "RemoveContainer" containerID="018d19ebd6b4e63cf62abd87c5492b69b0f2698020598490f89afd71a1f533f7" Nov 02 22:15:21.721763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.721731 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:21.820501 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.820385 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca\") pod \"2d94ac34-a86a-4753-b434-606ec862fc3d\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " Nov 02 22:15:21.820501 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.820474 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4\") pod \"2d94ac34-a86a-4753-b434-606ec862fc3d\" (UID: \"2d94ac34-a86a-4753-b434-606ec862fc3d\") " Nov 02 22:15:21.820837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.820812 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca" pod "2d94ac34-a86a-4753-b434-606ec862fc3d" (UID: "2d94ac34-a86a-4753-b434-606ec862fc3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:21.822666 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.822644 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4" pod "2d94ac34-a86a-4753-b434-606ec862fc3d" (UID: "2d94ac34-a86a-4753-b434-606ec862fc3d"). InnerVolumeSpecName "kube-api-access-6c2p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:21.828425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.828400 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv"] Nov 02 22:15:21.831989 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.831970 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb0367-29894295-ktkgv"] Nov 02 22:15:21.925086 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.922797 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d94ac34-a86a-4753-b434-606ec862fc3d-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:21.925086 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.922850 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/2d94ac34-a86a-4753-b434-606ec862fc3d-kube-api-access-6c2p4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:21.962784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.958288 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz"] Nov 02 22:15:21.966690 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:21.965898 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10fdd5-29894295-6plbz"] Nov 02 22:15:22.565002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.564972 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" Nov 02 22:15:22.565002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.564982 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2" event={"ID":"2d94ac34-a86a-4753-b434-606ec862fc3d","Type":"ContainerDied","Data":"09e3de931cbdea09295f8c885269355d8b9bd9a8ec880914747d0c1b0ee3b2dd"} Nov 02 22:15:22.565540 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.565033 5091 scope.go:122] "RemoveContainer" containerID="fff5e8ab81ec5d7f1841de4affe890500e8615c8122251dd4374e60b1b89f060" Nov 02 22:15:22.732831 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.732792 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2"] Nov 02 22:15:22.738508 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.738476 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-75577e-29894295-97vj2"] Nov 02 22:15:22.982128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.982089 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5af250-15a5-436a-ac4c-66516346482f" path="/var/lib/kubelet/pods/0a5af250-15a5-436a-ac4c-66516346482f/volumes" Nov 02 22:15:22.982543 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.982528 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" path="/var/lib/kubelet/pods/2d94ac34-a86a-4753-b434-606ec862fc3d/volumes" Nov 02 22:15:22.982988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:22.982971 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" path="/var/lib/kubelet/pods/5ad892a2-1fad-4ce6-869c-29b464409490/volumes" Nov 02 22:15:23.955776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:23.955665 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv"] Nov 02 22:15:24.003721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:24.003691 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb8112-zbbsv"] Nov 02 22:15:24.477962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:24.477926 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5"] Nov 02 22:15:24.478228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:24.478162 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" containerID="cri-o://b82b97be47f234259ab4d7a154454934f7506cd32bb7275ceb75115a4bcc5c03" gracePeriod=30 Nov 02 22:15:24.976237 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:24.976207 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469d3c45-f298-4d7c-8ecb-16cf0d29b836" path="/var/lib/kubelet/pods/469d3c45-f298-4d7c-8ecb-16cf0d29b836/volumes" Nov 02 22:15:29.668168 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.668130 5091 generic.go:356] "Generic (PLEG): container finished" podID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerID="b82b97be47f234259ab4d7a154454934f7506cd32bb7275ceb75115a4bcc5c03" exitCode=0 Nov 02 22:15:29.668542 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.668200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" event={"ID":"6df05dfa-a092-4271-a600-96dc2ce787dc","Type":"ContainerDied","Data":"b82b97be47f234259ab4d7a154454934f7506cd32bb7275ceb75115a4bcc5c03"} Nov 02 22:15:29.738337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.738307 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 22:15:29.868855 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.868825 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.868926 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.868966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.868992 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869024 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869018 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869107 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:29.869251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869132 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869167 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869228 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869257 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:29.869479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869277 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:29.869479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869350 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs\") pod \"6df05dfa-a092-4271-a600-96dc2ce787dc\" (UID: \"6df05dfa-a092-4271-a600-96dc2ce787dc\") " Nov 02 22:15:29.869479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869398 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:29.869796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869772 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:29.869947 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869927 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.870020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869954 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.870020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869970 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.870020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.869986 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.870020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.870000 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df05dfa-a092-4271-a600-96dc2ce787dc-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.871337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.871298 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:29.871459 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.871374 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:29.871709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.871666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:15:29.871794 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.871712 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng" pod "6df05dfa-a092-4271-a600-96dc2ce787dc" (UID: "6df05dfa-a092-4271-a600-96dc2ce787dc"). InnerVolumeSpecName "kube-api-access-zx5ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:29.971912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.971870 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.971912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.971909 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx5ng\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-kube-api-access-zx5ng\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.972104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.971926 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:29.972104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:29.971944 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df05dfa-a092-4271-a600-96dc2ce787dc-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:30.572033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.572001 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75"] Nov 02 22:15:30.572237 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.572210 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" containerID="cri-o://1139b68c8dbf3474d516c880f16f29390573263fa86c4eee20a77f6d88b487c5" gracePeriod=30 Nov 02 22:15:30.726243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.725707 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" event={"ID":"6df05dfa-a092-4271-a600-96dc2ce787dc","Type":"ContainerDied","Data":"7559cc37f40bffd035c24ee301fe70ab783dad3e8981fb2ba9dcea1c3817981c"} Nov 02 22:15:30.726243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.725766 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5" Nov 02 22:15:30.726243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.725781 5091 scope.go:122] "RemoveContainer" containerID="b82b97be47f234259ab4d7a154454934f7506cd32bb7275ceb75115a4bcc5c03" Nov 02 22:15:30.890182 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.890142 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5"] Nov 02 22:15:30.893698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.893659 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-sgjk5"] Nov 02 22:15:30.977962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:30.977933 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" path="/var/lib/kubelet/pods/6df05dfa-a092-4271-a600-96dc2ce787dc/volumes" Nov 02 22:15:31.040238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:31.040206 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:15:31.040439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:31.040277 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:15:35.396055 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.396019 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6"] Nov 02 22:15:35.445465 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.445438 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-75577e-vjlq6"] Nov 02 22:15:35.769706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.769152 5091 generic.go:356] "Generic (PLEG): container finished" podID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerID="1139b68c8dbf3474d516c880f16f29390573263fa86c4eee20a77f6d88b487c5" exitCode=0 Nov 02 22:15:35.769706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.769274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" event={"ID":"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548","Type":"ContainerDied","Data":"1139b68c8dbf3474d516c880f16f29390573263fa86c4eee20a77f6d88b487c5"} Nov 02 22:15:35.837683 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.837646 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 22:15:35.856988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.856963 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857108 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857005 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857108 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857108 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857114 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857160 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857185 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.857515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.857515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857327 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857381 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket\") pod \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\" (UID: \"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548\") " Nov 02 22:15:35.857515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857492 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.857773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857595 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:35.857843 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857799 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.857961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857906 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.857961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857932 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.857961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857948 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.858229 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857979 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.858229 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.857993 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.859488 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.859465 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "kube-api-access-qtb9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:35.859578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.859497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:35.859772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.859747 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:15:35.859864 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.859752 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy" pod "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" (UID: "cfcdf4cc-c621-4f1e-9ade-144ecd5c3548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.958870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.958804 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.958870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.958835 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.958870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.958848 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.958870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:35.958862 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtb9q\" (UniqueName: \"kubernetes.io/projected/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548-kube-api-access-qtb9q\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:36.777372 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:36.777343 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" Nov 02 22:15:36.777828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:36.777379 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75" event={"ID":"cfcdf4cc-c621-4f1e-9ade-144ecd5c3548","Type":"ContainerDied","Data":"ae2b412a1d0fb2c0747167985864ae31cf1bc21e444bd6205ec343246ad4d48b"} Nov 02 22:15:36.777828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:36.777437 5091 scope.go:122] "RemoveContainer" containerID="1139b68c8dbf3474d516c880f16f29390573263fa86c4eee20a77f6d88b487c5" Nov 02 22:15:36.977161 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:36.977129 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1866a79a-b747-4d9f-95fc-2edd8ce1b801" path="/var/lib/kubelet/pods/1866a79a-b747-4d9f-95fc-2edd8ce1b801/volumes" Nov 02 22:15:36.994772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:36.994743 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75"] Nov 02 22:15:37.043779 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:37.043713 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-rtw75"] Nov 02 22:15:37.112624 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:37.112591 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:15:38.976593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:38.976565 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" path="/var/lib/kubelet/pods/cfcdf4cc-c621-4f1e-9ade-144ecd5c3548/volumes" Nov 02 22:15:42.979113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979082 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp"] Nov 02 22:15:42.979546 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979534 5091 cpu_manager.go:414] "containerMap: removing container" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979547 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979561 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979566 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979574 5091 cpu_manager.go:414] "containerMap: removing container" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979579 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979587 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979592 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979600 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.979610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979606 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979615 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979620 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979627 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979632 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979639 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979643 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979649 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979654 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979659 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979663 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979688 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979695 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979710 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ca119-617d-4312-9f91-59faea5cc71f" containerName="revoke-keys" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979716 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ca119-617d-4312-9f91-59faea5cc71f" containerName="revoke-keys" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979724 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979729 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979739 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979744 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979754 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979759 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979765 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979770 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979777 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979782 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979788 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979792 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979801 5091 cpu_manager.go:414] "containerMap: removing container" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979805 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979812 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979816 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979823 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979827 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979837 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979841 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:15:42.979911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979923 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979931 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979939 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979945 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979949 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcdf4cc-c621-4f1e-9ade-144ecd5c3548" containerName="istio-proxy" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979956 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df05dfa-a092-4271-a600-96dc2ce787dc" containerName="istio-proxy" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979961 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979966 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979971 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979976 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979981 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979986 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979992 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.979998 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980003 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980008 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c800d43-835a-4235-8bac-196c3eec8b94" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980014 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980020 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ca119-617d-4312-9f91-59faea5cc71f" containerName="revoke-keys" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980026 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980030 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:15:42.980967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.980037 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe46fe00-01c6-4c24-aa59-0156502f45f2" containerName="cleanup" Nov 02 22:15:42.983348 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:42.983332 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.040255 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.040221 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp"] Nov 02 22:15:43.050963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.050929 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.051117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.051039 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.051117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.051065 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ns2\" (UniqueName: \"kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.085317 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.085286 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:43.085317 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.085316 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:15:43.085515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.085296 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:15:43.085515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.085362 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:15:43.152534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.152504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.152751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.152559 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.152751 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.152583 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ns2\" (UniqueName: \"kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.153262 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.153235 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.154892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.154872 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.208579 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.208556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ns2\" (UniqueName: \"kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2\") pod \"maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.296689 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.296615 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:43.684464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.684434 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp"] Nov 02 22:15:43.685084 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:15:43.685063 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760b945a_c7f9_4e5c_a6aa_05dd8587eac9.slice/crio-933483645ace0753a5a16f50228a8c104336297399593099b4f0d311d4eeafaa WatchSource:0}: Error finding container 933483645ace0753a5a16f50228a8c104336297399593099b4f0d311d4eeafaa: Status 404 returned error can't find the container with id 933483645ace0753a5a16f50228a8c104336297399593099b4f0d311d4eeafaa Nov 02 22:15:43.834184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:43.834151 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" event={"ID":"760b945a-c7f9-4e5c-a6aa-05dd8587eac9","Type":"ContainerStarted","Data":"933483645ace0753a5a16f50228a8c104336297399593099b4f0d311d4eeafaa"} Nov 02 22:15:44.843395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:44.843295 5091 generic.go:356] "Generic (PLEG): container finished" podID="760b945a-c7f9-4e5c-a6aa-05dd8587eac9" containerID="427e510c1d485299f33fd48da66381a876775330f924da19bd409e4b21903ee3" exitCode=0 Nov 02 22:15:44.843395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:44.843382 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" event={"ID":"760b945a-c7f9-4e5c-a6aa-05dd8587eac9","Type":"ContainerDied","Data":"427e510c1d485299f33fd48da66381a876775330f924da19bd409e4b21903ee3"} Nov 02 22:15:45.998902 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:45.998875 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:46.085022 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.084993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca\") pod \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " Nov 02 22:15:46.085215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.085063 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2\" (UniqueName: \"kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2\") pod \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " Nov 02 22:15:46.085215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.085154 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca\") pod \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\" (UID: \"760b945a-c7f9-4e5c-a6aa-05dd8587eac9\") " Nov 02 22:15:46.085525 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.085488 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca" pod "760b945a-c7f9-4e5c-a6aa-05dd8587eac9" (UID: "760b945a-c7f9-4e5c-a6aa-05dd8587eac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:46.087197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.087171 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca" pod "760b945a-c7f9-4e5c-a6aa-05dd8587eac9" (UID: "760b945a-c7f9-4e5c-a6aa-05dd8587eac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:46.087291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.087235 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2" pod "760b945a-c7f9-4e5c-a6aa-05dd8587eac9" (UID: "760b945a-c7f9-4e5c-a6aa-05dd8587eac9"). InnerVolumeSpecName "kube-api-access-l2ns2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:46.186281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.186247 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:46.186281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.186283 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:46.186488 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.186298 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ns2\" (UniqueName: \"kubernetes.io/projected/760b945a-c7f9-4e5c-a6aa-05dd8587eac9-kube-api-access-l2ns2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:46.860874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.860846 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" Nov 02 22:15:46.861129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.860878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp" event={"ID":"760b945a-c7f9-4e5c-a6aa-05dd8587eac9","Type":"ContainerDied","Data":"933483645ace0753a5a16f50228a8c104336297399593099b4f0d311d4eeafaa"} Nov 02 22:15:46.861291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:46.861162 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="933483645ace0753a5a16f50228a8c104336297399593099b4f0d311d4eeafaa" Nov 02 22:15:47.465217 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:47.465181 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427"] Nov 02 22:15:53.289730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:53.289688 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66"] Nov 02 22:15:53.290140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:53.289878 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" containerID="cri-o://fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f" gracePeriod=30 Nov 02 22:15:53.309545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:53.309514 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:15:53.309691 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:53.309571 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:15:55.537095 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:55.537049 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm"] Nov 02 22:15:55.641382 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:55.641352 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6"] Nov 02 22:15:58.548156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.548129 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 22:15:58.642997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.642970 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643007 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643091 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643136 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643187 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643216 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643262 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643294 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:58.643399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo\") pod \"c923486f-7415-4bf1-84f8-0dd19283d67a\" (UID: \"c923486f-7415-4bf1-84f8-0dd19283d67a\") " Nov 02 22:15:58.643872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:58.643872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643524 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:58.643872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643530 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:58.643872 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643541 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:58.644117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643929 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.644117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643950 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c923486f-7415-4bf1-84f8-0dd19283d67a-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.644117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643959 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.644117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643968 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.644117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.643977 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.645449 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.645425 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:58.645567 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.645535 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:58.645617 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.645602 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:15:58.645716 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.645698 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5" pod "c923486f-7415-4bf1-84f8-0dd19283d67a" (UID: "c923486f-7415-4bf1-84f8-0dd19283d67a"). InnerVolumeSpecName "kube-api-access-vpft5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:58.745770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.745735 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpft5\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-kube-api-access-vpft5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.745770 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.745768 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.745961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.745778 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:58.745961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:58.745789 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c923486f-7415-4bf1-84f8-0dd19283d67a-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:59.040602 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.040566 5091 generic.go:356] "Generic (PLEG): container finished" podID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerID="fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f" exitCode=0 Nov 02 22:15:59.040776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.040686 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" event={"ID":"c923486f-7415-4bf1-84f8-0dd19283d67a","Type":"ContainerDied","Data":"fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f"} Nov 02 22:15:59.040776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.040744 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" event={"ID":"c923486f-7415-4bf1-84f8-0dd19283d67a","Type":"ContainerDied","Data":"c9a5ed75c646cec6d21fe54cc8645aec9e26625f4a5ede0f67594a519b9c652b"} Nov 02 22:15:59.040880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.040776 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66" Nov 02 22:15:59.040880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.040775 5091 scope.go:122] "RemoveContainer" containerID="fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f" Nov 02 22:15:59.051830 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.051814 5091 scope.go:122] "RemoveContainer" containerID="fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f" Nov 02 22:15:59.052146 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:15:59.052125 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f\": container with ID starting with fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f not found: ID does not exist" containerID="fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f" Nov 02 22:15:59.052194 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.052156 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f"} err="failed to get container status \"fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f\": rpc error: code = NotFound desc = could not find container \"fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f\": container with ID starting with fd144f201794a64813c56b38b1304fa9011c352bc8e5639a350db1e1a1e1b99f not found: ID does not exist" Nov 02 22:15:59.204809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.204776 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66"] Nov 02 22:15:59.254581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:15:59.254511 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-6hx66"] Nov 02 22:16:00.537783 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.537725 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" podUID="85c5a543-df95-42a4-8646-05e69bef7e43" containerName="payload-processing" containerID="cri-o://38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832" gracePeriod=25 Nov 02 22:16:00.642632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.642594 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" podUID="eaa92439-8d82-4647-8e08-4b9192d4bb9d" containerName="payload-pre-processing" containerID="cri-o://070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29" gracePeriod=25 Nov 02 22:16:00.817425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.815754 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 22:16:00.870907 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.870877 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume\") pod \"85c5a543-df95-42a4-8646-05e69bef7e43\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " Nov 02 22:16:00.871104 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.870941 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf\" (UniqueName: \"kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf\") pod \"85c5a543-df95-42a4-8646-05e69bef7e43\" (UID: \"85c5a543-df95-42a4-8646-05e69bef7e43\") " Nov 02 22:16:00.871268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.871245 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume" pod "85c5a543-df95-42a4-8646-05e69bef7e43" (UID: "85c5a543-df95-42a4-8646-05e69bef7e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:00.872997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.872976 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf" pod "85c5a543-df95-42a4-8646-05e69bef7e43" (UID: "85c5a543-df95-42a4-8646-05e69bef7e43"). InnerVolumeSpecName "kube-api-access-wc9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:00.891444 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.891425 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 22:16:00.971875 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.971845 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume\") pod \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " Nov 02 22:16:00.972053 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.971977 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78\" (UniqueName: \"kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78\") pod \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\" (UID: \"eaa92439-8d82-4647-8e08-4b9192d4bb9d\") " Nov 02 22:16:00.972261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.972239 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume" pod "eaa92439-8d82-4647-8e08-4b9192d4bb9d" (UID: "eaa92439-8d82-4647-8e08-4b9192d4bb9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:00.972423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.972403 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa92439-8d82-4647-8e08-4b9192d4bb9d-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:00.972555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.972429 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c5a543-df95-42a4-8646-05e69bef7e43-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:00.972555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.972440 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9nf\" (UniqueName: \"kubernetes.io/projected/85c5a543-df95-42a4-8646-05e69bef7e43-kube-api-access-wc9nf\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:00.975875 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.974338 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78" pod "eaa92439-8d82-4647-8e08-4b9192d4bb9d" (UID: "eaa92439-8d82-4647-8e08-4b9192d4bb9d"). InnerVolumeSpecName "kube-api-access-dhx78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:00.985743 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:00.985718 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" path="/var/lib/kubelet/pods/c923486f-7415-4bf1-84f8-0dd19283d67a/volumes" Nov 02 22:16:01.057959 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.057876 5091 generic.go:356] "Generic (PLEG): container finished" podID="eaa92439-8d82-4647-8e08-4b9192d4bb9d" containerID="070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29" exitCode=0 Nov 02 22:16:01.058130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.057984 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" Nov 02 22:16:01.058130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.058009 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" event={"ID":"eaa92439-8d82-4647-8e08-4b9192d4bb9d","Type":"ContainerDied","Data":"070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29"} Nov 02 22:16:01.058130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.058058 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6" event={"ID":"eaa92439-8d82-4647-8e08-4b9192d4bb9d","Type":"ContainerDied","Data":"b192635e6fcf46bac08f37c590f9a1d5199c0efc88cbab597c1ee2e18f1753b0"} Nov 02 22:16:01.058130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.058089 5091 scope.go:122] "RemoveContainer" containerID="070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29" Nov 02 22:16:01.059869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.059834 5091 generic.go:356] "Generic (PLEG): container finished" podID="85c5a543-df95-42a4-8646-05e69bef7e43" containerID="38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832" exitCode=0 Nov 02 22:16:01.059999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.059894 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" Nov 02 22:16:01.060069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.060005 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" event={"ID":"85c5a543-df95-42a4-8646-05e69bef7e43","Type":"ContainerDied","Data":"38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832"} Nov 02 22:16:01.060069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.060046 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm" event={"ID":"85c5a543-df95-42a4-8646-05e69bef7e43","Type":"ContainerDied","Data":"73ff41eace52c07700213b359a5ca32aeb8ea0a0dc48a7e8b6e8867cb6c0979f"} Nov 02 22:16:01.069408 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.069381 5091 scope.go:122] "RemoveContainer" containerID="070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29" Nov 02 22:16:01.069712 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:16:01.069683 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29\": container with ID starting with 070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29 not found: ID does not exist" containerID="070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29" Nov 02 22:16:01.069803 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.069720 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29"} err="failed to get container status \"070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29\": rpc error: code = NotFound desc = could not find container \"070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29\": container with ID starting with 070faae562eef34f292cb328d23a4fcf2708e6f8367f9a8ee265f6c829865b29 not found: ID does not exist" Nov 02 22:16:01.069803 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.069739 5091 scope.go:122] "RemoveContainer" containerID="38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832" Nov 02 22:16:01.073366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.073340 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx78\" (UniqueName: \"kubernetes.io/projected/eaa92439-8d82-4647-8e08-4b9192d4bb9d-kube-api-access-dhx78\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:01.080572 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.080550 5091 scope.go:122] "RemoveContainer" containerID="38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832" Nov 02 22:16:01.080916 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:16:01.080890 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832\": container with ID starting with 38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832 not found: ID does not exist" containerID="38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832" Nov 02 22:16:01.080993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.080928 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832"} err="failed to get container status \"38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832\": rpc error: code = NotFound desc = could not find container \"38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832\": container with ID starting with 38f04b86e9e7a4f3a2d675ef3049b263361d7227650aefd88d7f38f985e9b832 not found: ID does not exist" Nov 02 22:16:01.222013 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.221978 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm"] Nov 02 22:16:01.226438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.226413 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-96c052-5c94b4d9cf-8z7pm"] Nov 02 22:16:01.333991 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.333909 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6"] Nov 02 22:16:01.338979 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:01.338953 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-96c052-65546f8cf9-ww4p6"] Nov 02 22:16:02.486582 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.486544 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" containerID="cri-o://571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a" gracePeriod=45 Nov 02 22:16:02.756126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.756102 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 22:16:02.888954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.888922 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache\") pod \"6a1d46d9-4610-49b7-a708-7087610aff3b\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " Nov 02 22:16:02.889123 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889024 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home\") pod \"6a1d46d9-4610-49b7-a708-7087610aff3b\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " Nov 02 22:16:02.889123 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889046 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm\") pod \"6a1d46d9-4610-49b7-a708-7087610aff3b\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " Nov 02 22:16:02.889227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889163 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n\" (UniqueName: \"kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n\") pod \"6a1d46d9-4610-49b7-a708-7087610aff3b\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " Nov 02 22:16:02.889227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889191 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir\") pod \"6a1d46d9-4610-49b7-a708-7087610aff3b\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " Nov 02 22:16:02.889227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889198 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache" pod "6a1d46d9-4610-49b7-a708-7087610aff3b" (UID: "6a1d46d9-4610-49b7-a708-7087610aff3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.889227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889222 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs\") pod \"6a1d46d9-4610-49b7-a708-7087610aff3b\" (UID: \"6a1d46d9-4610-49b7-a708-7087610aff3b\") " Nov 02 22:16:02.889402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889323 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home" pod "6a1d46d9-4610-49b7-a708-7087610aff3b" (UID: "6a1d46d9-4610-49b7-a708-7087610aff3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.889466 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889448 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir" pod "6a1d46d9-4610-49b7-a708-7087610aff3b" (UID: "6a1d46d9-4610-49b7-a708-7087610aff3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.889633 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889612 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.889788 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889653 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.889788 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.889686 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.891444 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.891423 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm" pod "6a1d46d9-4610-49b7-a708-7087610aff3b" (UID: "6a1d46d9-4610-49b7-a708-7087610aff3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.891557 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.891474 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n" pod "6a1d46d9-4610-49b7-a708-7087610aff3b" (UID: "6a1d46d9-4610-49b7-a708-7087610aff3b"). InnerVolumeSpecName "kube-api-access-4zf9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:02.891557 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.891487 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs" pod "6a1d46d9-4610-49b7-a708-7087610aff3b" (UID: "6a1d46d9-4610-49b7-a708-7087610aff3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:02.980787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.980748 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c5a543-df95-42a4-8646-05e69bef7e43" path="/var/lib/kubelet/pods/85c5a543-df95-42a4-8646-05e69bef7e43/volumes" Nov 02 22:16:02.981428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.981401 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa92439-8d82-4647-8e08-4b9192d4bb9d" path="/var/lib/kubelet/pods/eaa92439-8d82-4647-8e08-4b9192d4bb9d/volumes" Nov 02 22:16:02.990569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.990546 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1d46d9-4610-49b7-a708-7087610aff3b-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.990569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.990570 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zf9n\" (UniqueName: \"kubernetes.io/projected/6a1d46d9-4610-49b7-a708-7087610aff3b-kube-api-access-4zf9n\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.990784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:02.990580 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d46d9-4610-49b7-a708-7087610aff3b-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:03.080595 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.080559 5091 generic.go:356] "Generic (PLEG): container finished" podID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerID="571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a" exitCode=0 Nov 02 22:16:03.080595 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.080596 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" Nov 02 22:16:03.080885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.080636 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" event={"ID":"6a1d46d9-4610-49b7-a708-7087610aff3b","Type":"ContainerDied","Data":"571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a"} Nov 02 22:16:03.080885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.080723 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427" event={"ID":"6a1d46d9-4610-49b7-a708-7087610aff3b","Type":"ContainerDied","Data":"de448575f5b8f19cb7721ca325fe1e5cb046c86f6205be0a64396e30f80e95d0"} Nov 02 22:16:03.080885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.080753 5091 scope.go:122] "RemoveContainer" containerID="571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a" Nov 02 22:16:03.093877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.093853 5091 scope.go:122] "RemoveContainer" containerID="571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a" Nov 02 22:16:03.094197 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:16:03.094174 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a\": container with ID starting with 571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a not found: ID does not exist" containerID="571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a" Nov 02 22:16:03.094322 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.094213 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a"} err="failed to get container status \"571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a\": rpc error: code = NotFound desc = could not find container \"571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a\": container with ID starting with 571b81e0b6b903c858774c29e1b7313ee2423f197ac5b03e30dc48a3202c5c9a not found: ID does not exist" Nov 02 22:16:03.243593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.243512 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427"] Nov 02 22:16:03.249007 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:03.248983 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-w5-96c052-kserve-5c5c699d7-sz427"] Nov 02 22:16:04.979307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:04.979275 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" path="/var/lib/kubelet/pods/6a1d46d9-4610-49b7-a708-7087610aff3b/volumes" Nov 02 22:16:06.610816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:06.610782 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb"] Nov 02 22:16:06.658760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:06.658728 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ffe386-v9jqb"] Nov 02 22:16:06.976700 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:06.976653 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb19f9ba-a147-4d83-9edf-99c569468d3e" path="/var/lib/kubelet/pods/eb19f9ba-a147-4d83-9edf-99c569468d3e/volumes" Nov 02 22:16:07.112812 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:16:07.112771 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:16:13.643018 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:13.642987 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb"] Nov 02 22:16:13.643596 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:13.643244 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" containerID="cri-o://8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8" gracePeriod=30 Nov 02 22:16:18.904911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:18.904882 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 22:16:19.059797 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059728 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.059797 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059762 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059831 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059852 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059875 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059913 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.059982 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060016 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060064 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token\") pod \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\" (UID: \"f534c2d6-402a-45e4-8b5a-56e91520bdfc\") " Nov 02 22:16:19.060369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060049 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:19.060490 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060197 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:19.060490 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060255 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:19.060490 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:19.060490 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060304 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:19.060750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060575 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.060750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060597 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.060750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060612 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.060750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060627 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.060750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.060641 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.062131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.062099 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:19.062267 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.062177 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "kube-api-access-d6626". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:19.062333 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.062289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:16:19.062389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.062365 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token" pod "f534c2d6-402a-45e4-8b5a-56e91520bdfc" (UID: "f534c2d6-402a-45e4-8b5a-56e91520bdfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:19.162090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.162056 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.162090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.162084 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.162090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.162095 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6626\" (UniqueName: \"kubernetes.io/projected/f534c2d6-402a-45e4-8b5a-56e91520bdfc-kube-api-access-d6626\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.162322 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.162104 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f534c2d6-402a-45e4-8b5a-56e91520bdfc-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:19.249899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.249859 5091 generic.go:356] "Generic (PLEG): container finished" podID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerID="8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8" exitCode=0 Nov 02 22:16:19.250083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.249915 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" Nov 02 22:16:19.250083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.249943 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" event={"ID":"f534c2d6-402a-45e4-8b5a-56e91520bdfc","Type":"ContainerDied","Data":"8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8"} Nov 02 22:16:19.250083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.250001 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb" event={"ID":"f534c2d6-402a-45e4-8b5a-56e91520bdfc","Type":"ContainerDied","Data":"45aedd96597d91da8b984bb44dacb554455b63969ff70675e6f5809550df267d"} Nov 02 22:16:19.250083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.250027 5091 scope.go:122] "RemoveContainer" containerID="8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8" Nov 02 22:16:19.260967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.260950 5091 scope.go:122] "RemoveContainer" containerID="8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8" Nov 02 22:16:19.261241 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:16:19.261223 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8\": container with ID starting with 8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8 not found: ID does not exist" containerID="8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8" Nov 02 22:16:19.261283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.261250 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8"} err="failed to get container status \"8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8\": rpc error: code = NotFound desc = could not find container \"8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8\": container with ID starting with 8b9cff105ad47fb1010b08bc004dacc835d3c237e1176537cf70906ab67f3eb8 not found: ID does not exist" Nov 02 22:16:19.416431 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.416404 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb"] Nov 02 22:16:19.419285 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:19.419262 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-rhtxb"] Nov 02 22:16:20.571535 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:20.571499 5091 scope.go:122] "RemoveContainer" containerID="a3f096f50904f4e4df477b839cd604d48410bb41a9d40c36b2b5d56dda1b77bf" Nov 02 22:16:20.583908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:20.583880 5091 scope.go:122] "RemoveContainer" containerID="1e00140b948d84c6599b0ec986fd73ba17f0d688152b86ba29ac02120cc1e501" Nov 02 22:16:20.975727 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:20.975697 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" path="/var/lib/kubelet/pods/f534c2d6-402a-45e4-8b5a-56e91520bdfc/volumes" Nov 02 22:16:37.113222 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:16:37.113183 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:16:43.751829 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.751788 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k"] Nov 02 22:16:43.752272 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752258 5091 cpu_manager.go:414] "containerMap: removing container" podUID="eaa92439-8d82-4647-8e08-4b9192d4bb9d" containerName="payload-pre-processing" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752273 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa92439-8d82-4647-8e08-4b9192d4bb9d" containerName="payload-pre-processing" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752293 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752298 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752305 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752310 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5af250-15a5-436a-ac4c-66516346482f" containerName="cleanup" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752318 5091 cpu_manager.go:414] "containerMap: removing container" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:16:43.752321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752323 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752330 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752336 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752346 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752351 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752360 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752364 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752372 5091 cpu_manager.go:414] "containerMap: removing container" podUID="760b945a-c7f9-4e5c-a6aa-05dd8587eac9" containerName="revoke-keys" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752376 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760b945a-c7f9-4e5c-a6aa-05dd8587eac9" containerName="revoke-keys" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752382 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752386 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752396 5091 cpu_manager.go:414] "containerMap: removing container" podUID="85c5a543-df95-42a4-8646-05e69bef7e43" containerName="payload-processing" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752401 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c5a543-df95-42a4-8646-05e69bef7e43" containerName="payload-processing" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752477 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c5a543-df95-42a4-8646-05e69bef7e43" containerName="payload-processing" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752487 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d412b8e7-d825-40e3-9ccf-658184e53973" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752495 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad892a2-1fad-4ce6-869c-29b464409490" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752503 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa92439-8d82-4647-8e08-4b9192d4bb9d" containerName="payload-pre-processing" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752509 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6944953f-a3de-4004-a4f3-9f748d66d1c5" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752515 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d94ac34-a86a-4753-b434-606ec862fc3d" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752522 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f534c2d6-402a-45e4-8b5a-56e91520bdfc" containerName="istio-proxy" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752527 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="810cc234-d11c-4aa8-8754-9e7e3f36d571" containerName="cleanup" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752533 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1d46d9-4610-49b7-a708-7087610aff3b" containerName="main" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752538 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c923486f-7415-4bf1-84f8-0dd19283d67a" containerName="istio-proxy" Nov 02 22:16:43.752568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.752542 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="760b945a-c7f9-4e5c-a6aa-05dd8587eac9" containerName="revoke-keys" Nov 02 22:16:43.755802 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.755784 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.808858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.808827 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v"] Nov 02 22:16:43.810126 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.810098 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.810259 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.810171 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.810347 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.810285 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnglm\" (UniqueName: \"kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.816311 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.816290 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k"] Nov 02 22:16:43.816445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.816373 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:43.855645 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.855614 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:16:43.855794 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.855696 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:16:43.855794 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.855754 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:16:43.859366 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.859349 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:16:43.869183 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.869162 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v"] Nov 02 22:16:43.911845 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.911815 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zp7\" (UniqueName: \"kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:43.911995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.911867 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:43.911995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.911971 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.912090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.912030 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.912090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.912058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnglm\" (UniqueName: \"kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.912090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.912082 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:43.912603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.912580 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.914418 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.914398 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:43.968689 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:43.968647 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnglm\" (UniqueName: \"kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm\") pod \"maas-api-revoke-sub-6d8006d2-jw26k\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:44.013711 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.013611 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.013876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.013725 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.013876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.013795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zp7\" (UniqueName: \"kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.014433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.014413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.016029 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.015995 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.068745 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.068711 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:44.069659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.069637 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zp7\" (UniqueName: \"kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7\") pod \"maas-api-revoke-sub-574a4459-9v78v\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.133266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.133017 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:44.444078 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.444038 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k"] Nov 02 22:16:44.444904 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:16:44.444880 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf21d29_2db2_4da0_9ac0_4277b385af3e.slice/crio-0287141151a29b2651531b170ba670e8111ae0bc6f7e5a77a9b8d88543352e5f WatchSource:0}: Error finding container 0287141151a29b2651531b170ba670e8111ae0bc6f7e5a77a9b8d88543352e5f: Status 404 returned error can't find the container with id 0287141151a29b2651531b170ba670e8111ae0bc6f7e5a77a9b8d88543352e5f Nov 02 22:16:44.544631 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:44.544603 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v"] Nov 02 22:16:44.545986 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:16:44.545958 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966f4d6d_e979_4964_9a64_3eaa0b446bc5.slice/crio-71c102180316d0f77303f44efe68d6cd878e13b352cd7a4e8e66bf1f5215cf43 WatchSource:0}: Error finding container 71c102180316d0f77303f44efe68d6cd878e13b352cd7a4e8e66bf1f5215cf43: Status 404 returned error can't find the container with id 71c102180316d0f77303f44efe68d6cd878e13b352cd7a4e8e66bf1f5215cf43 Nov 02 22:16:45.434479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:45.434447 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" event={"ID":"966f4d6d-e979-4964-9a64-3eaa0b446bc5","Type":"ContainerStarted","Data":"59e10a82ec47c1ce79c7b8cb1196856197f2aa1c6f820e5eeb4de3df270daee8"} Nov 02 22:16:45.434953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:45.434507 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" event={"ID":"966f4d6d-e979-4964-9a64-3eaa0b446bc5","Type":"ContainerStarted","Data":"71c102180316d0f77303f44efe68d6cd878e13b352cd7a4e8e66bf1f5215cf43"} Nov 02 22:16:45.436396 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:45.436366 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" event={"ID":"1bf21d29-2db2-4da0-9ac0-4277b385af3e","Type":"ContainerStarted","Data":"2ca8c2673817f55bd5f6b3bdc7e3320cbb96be7bae979dc6c38995857c075112"} Nov 02 22:16:45.436505 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:45.436421 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" event={"ID":"1bf21d29-2db2-4da0-9ac0-4277b385af3e","Type":"ContainerStarted","Data":"0287141151a29b2651531b170ba670e8111ae0bc6f7e5a77a9b8d88543352e5f"} Nov 02 22:16:45.660250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:45.660196 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" podStartSLOduration=2.660181992 podStartE2EDuration="2.660181992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:45.658135606 +0000 UTC m=+2552.820230726" watchObservedRunningTime="2026-11-02 22:16:45.660181992 +0000 UTC m=+2552.822277112" Nov 02 22:16:46.449636 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.449597 5091 generic.go:356] "Generic (PLEG): container finished" podID="1bf21d29-2db2-4da0-9ac0-4277b385af3e" containerID="2ca8c2673817f55bd5f6b3bdc7e3320cbb96be7bae979dc6c38995857c075112" exitCode=0 Nov 02 22:16:46.450093 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.449694 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" event={"ID":"1bf21d29-2db2-4da0-9ac0-4277b385af3e","Type":"ContainerDied","Data":"2ca8c2673817f55bd5f6b3bdc7e3320cbb96be7bae979dc6c38995857c075112"} Nov 02 22:16:46.451243 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.451222 5091 generic.go:356] "Generic (PLEG): container finished" podID="966f4d6d-e979-4964-9a64-3eaa0b446bc5" containerID="59e10a82ec47c1ce79c7b8cb1196856197f2aa1c6f820e5eeb4de3df270daee8" exitCode=0 Nov 02 22:16:46.451331 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.451307 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" event={"ID":"966f4d6d-e979-4964-9a64-3eaa0b446bc5","Type":"ContainerDied","Data":"59e10a82ec47c1ce79c7b8cb1196856197f2aa1c6f820e5eeb4de3df270daee8"} Nov 02 22:16:46.594692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.594649 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:46.646242 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.646213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca\") pod \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " Nov 02 22:16:46.646425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.646304 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7\" (UniqueName: \"kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7\") pod \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " Nov 02 22:16:46.646425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.646346 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca\") pod \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\" (UID: \"966f4d6d-e979-4964-9a64-3eaa0b446bc5\") " Nov 02 22:16:46.646603 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.646577 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca" pod "966f4d6d-e979-4964-9a64-3eaa0b446bc5" (UID: "966f4d6d-e979-4964-9a64-3eaa0b446bc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:46.648369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.648341 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7" pod "966f4d6d-e979-4964-9a64-3eaa0b446bc5" (UID: "966f4d6d-e979-4964-9a64-3eaa0b446bc5"). InnerVolumeSpecName "kube-api-access-58zp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:46.648475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.648341 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca" pod "966f4d6d-e979-4964-9a64-3eaa0b446bc5" (UID: "966f4d6d-e979-4964-9a64-3eaa0b446bc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:46.747388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.747323 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:46.747388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.747348 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zp7\" (UniqueName: \"kubernetes.io/projected/966f4d6d-e979-4964-9a64-3eaa0b446bc5-kube-api-access-58zp7\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:46.747388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:46.747358 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966f4d6d-e979-4964-9a64-3eaa0b446bc5-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:47.458466 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.458438 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" Nov 02 22:16:47.458895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.458478 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v" event={"ID":"966f4d6d-e979-4964-9a64-3eaa0b446bc5","Type":"ContainerDied","Data":"71c102180316d0f77303f44efe68d6cd878e13b352cd7a4e8e66bf1f5215cf43"} Nov 02 22:16:47.458895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.458524 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c102180316d0f77303f44efe68d6cd878e13b352cd7a4e8e66bf1f5215cf43" Nov 02 22:16:47.597528 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.597495 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:47.658186 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.658156 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca\") pod \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " Nov 02 22:16:47.658318 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.658308 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm\" (UniqueName: \"kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm\") pod \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " Nov 02 22:16:47.658363 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.658333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca\") pod \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\" (UID: \"1bf21d29-2db2-4da0-9ac0-4277b385af3e\") " Nov 02 22:16:47.658796 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.658770 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca" pod "1bf21d29-2db2-4da0-9ac0-4277b385af3e" (UID: "1bf21d29-2db2-4da0-9ac0-4277b385af3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:47.660235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.660211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca" pod "1bf21d29-2db2-4da0-9ac0-4277b385af3e" (UID: "1bf21d29-2db2-4da0-9ac0-4277b385af3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:47.660344 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.660267 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm" pod "1bf21d29-2db2-4da0-9ac0-4277b385af3e" (UID: "1bf21d29-2db2-4da0-9ac0-4277b385af3e"). InnerVolumeSpecName "kube-api-access-dnglm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:47.759583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.759513 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnglm\" (UniqueName: \"kubernetes.io/projected/1bf21d29-2db2-4da0-9ac0-4277b385af3e-kube-api-access-dnglm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:47.759583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.759542 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:47.759583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:47.759555 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf21d29-2db2-4da0-9ac0-4277b385af3e-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:48.466165 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:48.466127 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" Nov 02 22:16:48.466553 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:48.466163 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k" event={"ID":"1bf21d29-2db2-4da0-9ac0-4277b385af3e","Type":"ContainerDied","Data":"0287141151a29b2651531b170ba670e8111ae0bc6f7e5a77a9b8d88543352e5f"} Nov 02 22:16:48.466553 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:16:48.466216 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0287141151a29b2651531b170ba670e8111ae0bc6f7e5a77a9b8d88543352e5f" Nov 02 22:17:07.111818 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:17:07.111779 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:17:20.678705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:20.678656 5091 scope.go:122] "RemoveContainer" containerID="43e19846384ce5cda96733c6ff5dbde027f6162a9380aa18610c350ab1ea74c5" Nov 02 22:17:20.688684 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:20.688652 5091 scope.go:122] "RemoveContainer" containerID="3eecc6be635b1c8e3cd5146c565629e952b66c2cded519e641eec0d1b52ef7bc" Nov 02 22:17:20.699545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:20.699520 5091 scope.go:122] "RemoveContainer" containerID="c24b9f3afc91adc420d3f0cb31fda5b84b01afc530cf6dbcdc370f26a6bf6b4b" Nov 02 22:17:33.466779 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:33.466741 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp"] Nov 02 22:17:33.515586 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:33.515549 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-96c052-pwqgp"] Nov 02 22:17:34.975785 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:34.975751 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="760b945a-c7f9-4e5c-a6aa-05dd8587eac9" path="/var/lib/kubelet/pods/760b945a-c7f9-4e5c-a6aa-05dd8587eac9/volumes" Nov 02 22:17:37.112735 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:17:37.112701 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:17:40.336964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:40.336926 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx"] Nov 02 22:17:40.337474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:40.337210 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" containerID="cri-o://240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f" gracePeriod=30 Nov 02 22:17:45.594230 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.594200 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 22:17:45.697581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.697789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697589 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.697789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.697789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697650 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.697789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697698 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.697789 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697781 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.698135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697918 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.698135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697930 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.698135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.697944 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.698135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698039 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.698135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698115 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.698335 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698240 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:45.698387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698351 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.698432 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698394 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token\") pod \"6ff668d6-1c08-4d97-882f-5d7e05372b88\" (UID: \"6ff668d6-1c08-4d97-882f-5d7e05372b88\") " Nov 02 22:17:45.698793 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698767 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.698895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698795 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff668d6-1c08-4d97-882f-5d7e05372b88-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.698895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698813 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.698895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698826 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.698895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.698841 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.700251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.700221 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:17:45.700371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.700292 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.700608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.700590 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:45.700711 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.700666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g" pod "6ff668d6-1c08-4d97-882f-5d7e05372b88" (UID: "6ff668d6-1c08-4d97-882f-5d7e05372b88"). InnerVolumeSpecName "kube-api-access-9d92g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:45.800195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.800165 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.800195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.800191 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.800195 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.800202 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d92g\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-kube-api-access-9d92g\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.800485 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.800211 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff668d6-1c08-4d97-882f-5d7e05372b88-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.847798 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.847763 5091 generic.go:356] "Generic (PLEG): container finished" podID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerID="240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f" exitCode=0 Nov 02 22:17:45.847993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.847821 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" Nov 02 22:17:45.847993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.847857 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" event={"ID":"6ff668d6-1c08-4d97-882f-5d7e05372b88","Type":"ContainerDied","Data":"240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f"} Nov 02 22:17:45.847993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.847912 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx" event={"ID":"6ff668d6-1c08-4d97-882f-5d7e05372b88","Type":"ContainerDied","Data":"8fd20a645fd52ee11e24d0bfa7b566af7811c2d66140084755e2355033e691d2"} Nov 02 22:17:45.847993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.847935 5091 scope.go:122] "RemoveContainer" containerID="240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f" Nov 02 22:17:45.859405 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.859389 5091 scope.go:122] "RemoveContainer" containerID="240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f" Nov 02 22:17:45.859684 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:17:45.859652 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f\": container with ID starting with 240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f not found: ID does not exist" containerID="240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f" Nov 02 22:17:45.859773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:45.859690 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f"} err="failed to get container status \"240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f\": rpc error: code = NotFound desc = could not find container \"240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f\": container with ID starting with 240e6630ef7b0061b485615c5cd4e1138cbf0728b7327b922344fdbf7900649f not found: ID does not exist" Nov 02 22:17:46.012387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:46.012303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx"] Nov 02 22:17:46.014927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:46.014904 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-zjcwx"] Nov 02 22:17:46.975644 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:17:46.975608 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" path="/var/lib/kubelet/pods/6ff668d6-1c08-4d97-882f-5d7e05372b88/volumes" Nov 02 22:18:07.112339 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:18:07.112291 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:18:11.574999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.574961 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz"] Nov 02 22:18:11.575428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575417 5091 cpu_manager.go:414] "containerMap: removing container" podUID="966f4d6d-e979-4964-9a64-3eaa0b446bc5" containerName="revoke-keys" Nov 02 22:18:11.575481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575428 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966f4d6d-e979-4964-9a64-3eaa0b446bc5" containerName="revoke-keys" Nov 02 22:18:11.575481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575451 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1bf21d29-2db2-4da0-9ac0-4277b385af3e" containerName="revoke-keys" Nov 02 22:18:11.575481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575456 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf21d29-2db2-4da0-9ac0-4277b385af3e" containerName="revoke-keys" Nov 02 22:18:11.575481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575462 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" Nov 02 22:18:11.575481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575467 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" Nov 02 22:18:11.575646 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575533 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf21d29-2db2-4da0-9ac0-4277b385af3e" containerName="revoke-keys" Nov 02 22:18:11.575646 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575542 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff668d6-1c08-4d97-882f-5d7e05372b88" containerName="istio-proxy" Nov 02 22:18:11.575646 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.575553 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="966f4d6d-e979-4964-9a64-3eaa0b446bc5" containerName="revoke-keys" Nov 02 22:18:11.578733 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.578717 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.635263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.635230 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz"] Nov 02 22:18:11.663829 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.663797 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.663986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.663866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.663986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.663916 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.663986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.663959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6dp4\" (UniqueName: \"kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.664093 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.663990 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.664129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.664081 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.680824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.680783 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-94a8a3\"/\"openshift-service-ca.crt\"" Nov 02 22:18:11.680982 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.680876 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-94a8a3\"/\"kube-root-ca.crt\"" Nov 02 22:18:11.680982 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.680876 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-94a8a3\"/\"default-dockercfg-5hqrw\"" Nov 02 22:18:11.681941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.681928 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-94a8a3\"/\"test-model-94a8a3-kserve-self-signed-certs\"" Nov 02 22:18:11.764917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.764882 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.764917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.764924 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765200 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765047 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6dp4\" (UniqueName: \"kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765200 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765096 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765200 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765137 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765253 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765408 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765450 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.765776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.765554 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.767256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.767234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.767684 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.767655 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.822469 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.822430 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6dp4\" (UniqueName: \"kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4\") pod \"test-model-94a8a3-kserve-5856784559-fnhmz\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:11.889951 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:11.889921 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:12.264506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:12.264462 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz"] Nov 02 22:18:12.266988 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:18:12.266956 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2add07da_f806_4412_89cb_43aa2a40f3d4.slice/crio-b7f7b3e50e99e072b86d106d03434f7670882442ab2482becc6952e96caf505f WatchSource:0}: Error finding container b7f7b3e50e99e072b86d106d03434f7670882442ab2482becc6952e96caf505f: Status 404 returned error can't find the container with id b7f7b3e50e99e072b86d106d03434f7670882442ab2482becc6952e96caf505f Nov 02 22:18:13.063475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:13.063433 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" event={"ID":"2add07da-f806-4412-89cb-43aa2a40f3d4","Type":"ContainerStarted","Data":"69903c391051a436c1dccf972f891a14b397ab270740f080e614876cfe09ef8a"} Nov 02 22:18:13.063876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:13.063485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" event={"ID":"2add07da-f806-4412-89cb-43aa2a40f3d4","Type":"ContainerStarted","Data":"b7f7b3e50e99e072b86d106d03434f7670882442ab2482becc6952e96caf505f"} Nov 02 22:18:13.178192 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:13.178144 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" podStartSLOduration=2.178129721 podStartE2EDuration="2.178129721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:13.176760571 +0000 UTC m=+2640.338855691" watchObservedRunningTime="2026-11-02 22:18:13.178129721 +0000 UTC m=+2640.340224835" Nov 02 22:18:13.890074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:13.890042 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:16.047548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:16.047512 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst"] Nov 02 22:18:16.097301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:16.097270 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms"] Nov 02 22:18:16.099901 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:16.099881 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7c62b3-7mhst"] Nov 02 22:18:16.102064 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:16.102046 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af481bea-9fhms"] Nov 02 22:18:16.976761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:16.976730 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c163b3-f8a5-485e-b316-abaf22ebdef9" path="/var/lib/kubelet/pods/24c163b3-f8a5-485e-b316-abaf22ebdef9/volumes" Nov 02 22:18:16.977166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:16.977153 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44628257-a764-488d-9128-8a86fa97d47c" path="/var/lib/kubelet/pods/44628257-a764-488d-9128-8a86fa97d47c/volumes" Nov 02 22:18:20.734840 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:20.734801 5091 scope.go:122] "RemoveContainer" containerID="d2b2533f051637fa3b33bbd14099a15507a90e7a003adeeab41c06082ad4e914" Nov 02 22:18:20.746212 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:20.746186 5091 scope.go:122] "RemoveContainer" containerID="a9a5946a3a009d0f7528b64e4bd0bde3c9da4f85bb461903918c352c11ab2a17" Nov 02 22:18:20.757106 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:20.757086 5091 scope.go:122] "RemoveContainer" containerID="909704827f646e8cb369227343d24156f0b82be96cd1a23cd68fa3b3af599551" Nov 02 22:18:21.890238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:21.890205 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:21.902666 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:21.902637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:22.135061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:22.135030 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:18:37.112080 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:18:37.112002 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:18:37.269772 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.269738 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2"] Nov 02 22:18:37.275395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.275374 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.331241 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.331211 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2"] Nov 02 22:18:37.375973 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.375940 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:37.375973 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.375968 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:37.376152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.375985 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:37.377168 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.377146 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:18:37.421557 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.421533 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.421666 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.421608 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.421741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.421694 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pw4\" (UniqueName: \"kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.523170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.523139 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.523309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.523180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pw4\" (UniqueName: \"kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.523309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.523245 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.523938 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.523917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.525462 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.525443 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.579186 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.579162 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pw4\" (UniqueName: \"kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4\") pod \"maas-api-revoke-sub-5416ece4-f9zj2\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.588560 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.588540 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:37.987648 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:37.987606 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2"] Nov 02 22:18:38.226682 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:38.226642 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" event={"ID":"0c6db464-1d01-42e0-8f73-76fa47919a9a","Type":"ContainerStarted","Data":"aae2382877a57144941e86e251c6ad98b872608393a60a6aafa02acc3fa849c6"} Nov 02 22:18:39.234291 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:39.234258 5091 generic.go:356] "Generic (PLEG): container finished" podID="0c6db464-1d01-42e0-8f73-76fa47919a9a" containerID="dca994a0de5325d02dccf877e24d2d71bf8cbe555243b0239082977e0f59c175" exitCode=0 Nov 02 22:18:39.234715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:39.234341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" event={"ID":"0c6db464-1d01-42e0-8f73-76fa47919a9a","Type":"ContainerDied","Data":"dca994a0de5325d02dccf877e24d2d71bf8cbe555243b0239082977e0f59c175"} Nov 02 22:18:40.385436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.385414 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:40.556495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.556418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca\") pod \"0c6db464-1d01-42e0-8f73-76fa47919a9a\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " Nov 02 22:18:40.556662 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.556627 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca\") pod \"0c6db464-1d01-42e0-8f73-76fa47919a9a\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " Nov 02 22:18:40.556755 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.556657 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4\" (UniqueName: \"kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4\") pod \"0c6db464-1d01-42e0-8f73-76fa47919a9a\" (UID: \"0c6db464-1d01-42e0-8f73-76fa47919a9a\") " Nov 02 22:18:40.556817 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.556767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca" pod "0c6db464-1d01-42e0-8f73-76fa47919a9a" (UID: "0c6db464-1d01-42e0-8f73-76fa47919a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:40.557042 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.557025 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:18:40.558616 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.558594 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca" pod "0c6db464-1d01-42e0-8f73-76fa47919a9a" (UID: "0c6db464-1d01-42e0-8f73-76fa47919a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:40.558733 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.558627 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4" pod "0c6db464-1d01-42e0-8f73-76fa47919a9a" (UID: "0c6db464-1d01-42e0-8f73-76fa47919a9a"). InnerVolumeSpecName "kube-api-access-62pw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:40.657919 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.657889 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6db464-1d01-42e0-8f73-76fa47919a9a-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:18:40.657919 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:40.657917 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pw4\" (UniqueName: \"kubernetes.io/projected/0c6db464-1d01-42e0-8f73-76fa47919a9a-kube-api-access-62pw4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:18:41.247990 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:41.247904 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" Nov 02 22:18:41.247990 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:41.247953 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2" event={"ID":"0c6db464-1d01-42e0-8f73-76fa47919a9a","Type":"ContainerDied","Data":"aae2382877a57144941e86e251c6ad98b872608393a60a6aafa02acc3fa849c6"} Nov 02 22:18:41.248224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:41.248007 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aae2382877a57144941e86e251c6ad98b872608393a60a6aafa02acc3fa849c6" Nov 02 22:18:50.293325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.293282 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt"] Nov 02 22:18:50.293986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.293762 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0c6db464-1d01-42e0-8f73-76fa47919a9a" containerName="revoke-keys" Nov 02 22:18:50.293986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.293776 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6db464-1d01-42e0-8f73-76fa47919a9a" containerName="revoke-keys" Nov 02 22:18:50.293986 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.293874 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6db464-1d01-42e0-8f73-76fa47919a9a" containerName="revoke-keys" Nov 02 22:18:50.297090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.297071 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.345393 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.345365 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt"] Nov 02 22:18:50.359120 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.359087 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.359248 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.359125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpt8\" (UniqueName: \"kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.359296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.359272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.391523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.391496 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:50.391523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.391508 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:50.391702 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.391498 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:50.398059 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.398037 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:18:50.459746 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.459718 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.459900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.459757 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpt8\" (UniqueName: \"kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.459900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.459829 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.460439 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.460414 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.462144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.462123 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.516475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.516451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpt8\" (UniqueName: \"kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8\") pod \"maas-api-revoke-sub-64b8f318-2djwt\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.612337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.612268 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:50.985107 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:50.985083 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt"] Nov 02 22:18:50.986753 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:18:50.986727 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792e6a6f_be29_40e8_8152_41146a05af4e.slice/crio-34d3b8b7d87c57b517deaa048db9428890a9d39939492f741a7086de936e0064 WatchSource:0}: Error finding container 34d3b8b7d87c57b517deaa048db9428890a9d39939492f741a7086de936e0064: Status 404 returned error can't find the container with id 34d3b8b7d87c57b517deaa048db9428890a9d39939492f741a7086de936e0064 Nov 02 22:18:51.315227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:51.315198 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" event={"ID":"792e6a6f-be29-40e8-8152-41146a05af4e","Type":"ContainerStarted","Data":"86b04913865b33dce87844e894c63773775a6116faa6416f8bdedd2e39b646de"} Nov 02 22:18:51.315597 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:51.315250 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" event={"ID":"792e6a6f-be29-40e8-8152-41146a05af4e","Type":"ContainerStarted","Data":"34d3b8b7d87c57b517deaa048db9428890a9d39939492f741a7086de936e0064"} Nov 02 22:18:51.430452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:51.430403 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" podStartSLOduration=1.430389329 podStartE2EDuration="1.430389329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:51.426946397 +0000 UTC m=+2678.589041529" watchObservedRunningTime="2026-11-02 22:18:51.430389329 +0000 UTC m=+2678.592484449" Nov 02 22:18:52.322843 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:52.322813 5091 generic.go:356] "Generic (PLEG): container finished" podID="792e6a6f-be29-40e8-8152-41146a05af4e" containerID="86b04913865b33dce87844e894c63773775a6116faa6416f8bdedd2e39b646de" exitCode=0 Nov 02 22:18:52.323221 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:52.322888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" event={"ID":"792e6a6f-be29-40e8-8152-41146a05af4e","Type":"ContainerDied","Data":"86b04913865b33dce87844e894c63773775a6116faa6416f8bdedd2e39b646de"} Nov 02 22:18:53.473943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.473920 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:53.484066 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.484044 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8\" (UniqueName: \"kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8\") pod \"792e6a6f-be29-40e8-8152-41146a05af4e\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " Nov 02 22:18:53.484185 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.484101 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca\") pod \"792e6a6f-be29-40e8-8152-41146a05af4e\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " Nov 02 22:18:53.484227 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.484210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca\") pod \"792e6a6f-be29-40e8-8152-41146a05af4e\" (UID: \"792e6a6f-be29-40e8-8152-41146a05af4e\") " Nov 02 22:18:53.484471 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.484446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca" pod "792e6a6f-be29-40e8-8152-41146a05af4e" (UID: "792e6a6f-be29-40e8-8152-41146a05af4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:53.484575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.484534 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:18:53.486078 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.486059 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8" pod "792e6a6f-be29-40e8-8152-41146a05af4e" (UID: "792e6a6f-be29-40e8-8152-41146a05af4e"). InnerVolumeSpecName "kube-api-access-wcpt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:53.486151 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.486124 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca" pod "792e6a6f-be29-40e8-8152-41146a05af4e" (UID: "792e6a6f-be29-40e8-8152-41146a05af4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:53.585209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.585113 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792e6a6f-be29-40e8-8152-41146a05af4e-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:18:53.585209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:53.585156 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpt8\" (UniqueName: \"kubernetes.io/projected/792e6a6f-be29-40e8-8152-41146a05af4e-kube-api-access-wcpt8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:18:54.337954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:54.337921 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" Nov 02 22:18:54.338155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:54.337962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt" event={"ID":"792e6a6f-be29-40e8-8152-41146a05af4e","Type":"ContainerDied","Data":"34d3b8b7d87c57b517deaa048db9428890a9d39939492f741a7086de936e0064"} Nov 02 22:18:54.338155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:18:54.338047 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d3b8b7d87c57b517deaa048db9428890a9d39939492f741a7086de936e0064" Nov 02 22:19:07.112632 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:19:07.112588 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:19:13.496268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:13.496237 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:19:13.502207 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:13.502185 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:19:16.046375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:16.046340 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm"] Nov 02 22:19:16.096089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:16.096059 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180a8a31-n7knm"] Nov 02 22:19:16.976423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:16.976392 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352f4baf-0508-4307-9d95-c903c98dc9ca" path="/var/lib/kubelet/pods/352f4baf-0508-4307-9d95-c903c98dc9ca/volumes" Nov 02 22:19:20.796354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:20.796321 5091 scope.go:122] "RemoveContainer" containerID="4c4311bb75d570b4b57721411b0e97cd77238ab0189eeedc69c89cd93ea8e571" Nov 02 22:19:20.806073 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:20.806047 5091 scope.go:122] "RemoveContainer" containerID="afd3d630d4d6596197c8f31439545e52cdeee0ba395cfb4e3a6dce45faed6d9d" Nov 02 22:19:20.814947 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:20.814927 5091 scope.go:122] "RemoveContainer" containerID="9f336ca8070497774e3fa45c1ebfe38cb7bd0d34796bbfa4dce4116e390e5541" Nov 02 22:19:20.823068 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:20.823051 5091 scope.go:122] "RemoveContainer" containerID="5559cd8c088b6cb12a92e07c1584c43a51d0081c2f47e6db4e85052d4aa3f695" Nov 02 22:19:20.831564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:20.831547 5091 scope.go:122] "RemoveContainer" containerID="ffb00204ee22650f659706b5c72276a3d1c9bedf7ebeb7597bcff2843adeb08b" Nov 02 22:19:20.842207 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:20.842187 5091 scope.go:122] "RemoveContainer" containerID="e7d8ceff5ec2b24c990a89d67fef9ecc0335cfe78c9dd0d03784a2f80d05508a" Nov 02 22:19:25.821655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.821621 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9"] Nov 02 22:19:25.822235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.822217 5091 cpu_manager.go:414] "containerMap: removing container" podUID="792e6a6f-be29-40e8-8152-41146a05af4e" containerName="revoke-keys" Nov 02 22:19:25.822297 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.822237 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792e6a6f-be29-40e8-8152-41146a05af4e" containerName="revoke-keys" Nov 02 22:19:25.822337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.822331 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="792e6a6f-be29-40e8-8152-41146a05af4e" containerName="revoke-keys" Nov 02 22:19:25.827827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.827807 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:25.882134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.882099 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9"] Nov 02 22:19:25.923774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.923743 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:25.923917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.923904 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zmf\" (UniqueName: \"kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:25.923960 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.923949 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:25.927883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.927858 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:25.928004 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.927953 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:19:25.929536 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.929515 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:25.929536 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:25.929513 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:26.024995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.024969 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zmf\" (UniqueName: \"kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.025131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.025022 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.025210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.025191 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.025700 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.025663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.027555 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.027535 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.080980 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.080926 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zmf\" (UniqueName: \"kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf\") pod \"maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.141382 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.141346 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:26.513899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.513872 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9"] Nov 02 22:19:26.515469 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:19:26.515438 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df78b07_70d3_4366_9eea_f4647a71f2bb.slice/crio-fa8d48d4f55727f2e2a00e374da3612d2eb547b7ece329e1cfbf4a66bafaec5a WatchSource:0}: Error finding container fa8d48d4f55727f2e2a00e374da3612d2eb547b7ece329e1cfbf4a66bafaec5a: Status 404 returned error can't find the container with id fa8d48d4f55727f2e2a00e374da3612d2eb547b7ece329e1cfbf4a66bafaec5a Nov 02 22:19:26.551573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:26.551551 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" event={"ID":"4df78b07-70d3-4366-9eea-f4647a71f2bb","Type":"ContainerStarted","Data":"fa8d48d4f55727f2e2a00e374da3612d2eb547b7ece329e1cfbf4a66bafaec5a"} Nov 02 22:19:27.045704 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:27.045622 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d"] Nov 02 22:19:27.048307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:27.048284 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a91e24c-kjr7d"] Nov 02 22:19:27.559369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:27.559339 5091 generic.go:356] "Generic (PLEG): container finished" podID="4df78b07-70d3-4366-9eea-f4647a71f2bb" containerID="73caac335457d689642cf211893ca39d551e6002ee7490691be34b07090865f4" exitCode=0 Nov 02 22:19:27.559545 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:27.559430 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" event={"ID":"4df78b07-70d3-4366-9eea-f4647a71f2bb","Type":"ContainerDied","Data":"73caac335457d689642cf211893ca39d551e6002ee7490691be34b07090865f4"} Nov 02 22:19:28.053791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.053756 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm"] Nov 02 22:19:28.058613 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.058587 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f95790-rpwjm"] Nov 02 22:19:28.708863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.708839 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:28.752281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.752255 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca\") pod \"4df78b07-70d3-4366-9eea-f4647a71f2bb\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " Nov 02 22:19:28.752427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.752381 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca\") pod \"4df78b07-70d3-4366-9eea-f4647a71f2bb\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " Nov 02 22:19:28.752427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.752409 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf\" (UniqueName: \"kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf\") pod \"4df78b07-70d3-4366-9eea-f4647a71f2bb\" (UID: \"4df78b07-70d3-4366-9eea-f4647a71f2bb\") " Nov 02 22:19:28.752696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.752646 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca" pod "4df78b07-70d3-4366-9eea-f4647a71f2bb" (UID: "4df78b07-70d3-4366-9eea-f4647a71f2bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:28.752791 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.752760 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:28.754328 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.754300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca" pod "4df78b07-70d3-4366-9eea-f4647a71f2bb" (UID: "4df78b07-70d3-4366-9eea-f4647a71f2bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:28.754419 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.754406 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf" pod "4df78b07-70d3-4366-9eea-f4647a71f2bb" (UID: "4df78b07-70d3-4366-9eea-f4647a71f2bb"). InnerVolumeSpecName "kube-api-access-p9zmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:28.853742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.853630 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zmf\" (UniqueName: \"kubernetes.io/projected/4df78b07-70d3-4366-9eea-f4647a71f2bb-kube-api-access-p9zmf\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:28.853742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.853658 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df78b07-70d3-4366-9eea-f4647a71f2bb-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:28.976370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.976336 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e124af-a6ec-4c9f-8a4f-7cb419ad7340" path="/var/lib/kubelet/pods/01e124af-a6ec-4c9f-8a4f-7cb419ad7340/volumes" Nov 02 22:19:28.976803 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:28.976789 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66263ea9-bb12-4968-83a6-ffc87473abb8" path="/var/lib/kubelet/pods/66263ea9-bb12-4968-83a6-ffc87473abb8/volumes" Nov 02 22:19:29.572446 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:29.572419 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" Nov 02 22:19:29.572892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:29.572460 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9" event={"ID":"4df78b07-70d3-4366-9eea-f4647a71f2bb","Type":"ContainerDied","Data":"fa8d48d4f55727f2e2a00e374da3612d2eb547b7ece329e1cfbf4a66bafaec5a"} Nov 02 22:19:29.572892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:29.572531 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa8d48d4f55727f2e2a00e374da3612d2eb547b7ece329e1cfbf4a66bafaec5a" Nov 02 22:19:34.050941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:34.050903 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5"] Nov 02 22:19:34.099900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:34.099862 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae7283f-2npd5"] Nov 02 22:19:34.976499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:34.976467 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b53642-1053-4cb4-9955-82ea50cc2d28" path="/var/lib/kubelet/pods/f0b53642-1053-4cb4-9955-82ea50cc2d28/volumes" Nov 02 22:19:35.059748 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:35.059715 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn"] Nov 02 22:19:35.109041 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:35.109009 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ed4a824-6s9xn"] Nov 02 22:19:36.976464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:36.976429 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9390ded7-fee7-4f13-853c-b849b94ba11e" path="/var/lib/kubelet/pods/9390ded7-fee7-4f13-853c-b849b94ba11e/volumes" Nov 02 22:19:37.112836 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:19:37.112803 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:19:42.995232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:42.995193 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9"] Nov 02 22:19:43.044608 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:43.044551 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1789c6-cvrr9"] Nov 02 22:19:44.976245 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:44.976208 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df78b07-70d3-4366-9eea-f4647a71f2bb" path="/var/lib/kubelet/pods/4df78b07-70d3-4366-9eea-f4647a71f2bb/volumes" Nov 02 22:19:50.876149 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.876109 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8"] Nov 02 22:19:50.876798 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.876776 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4df78b07-70d3-4366-9eea-f4647a71f2bb" containerName="revoke-keys" Nov 02 22:19:50.876909 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.876800 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df78b07-70d3-4366-9eea-f4647a71f2bb" containerName="revoke-keys" Nov 02 22:19:50.876988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.876974 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df78b07-70d3-4366-9eea-f4647a71f2bb" containerName="revoke-keys" Nov 02 22:19:50.880470 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.880449 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:50.934547 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.934527 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:19:50.934759 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.934744 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:50.984339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.984304 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:50.984339 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.984326 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:50.987088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.987069 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8"] Nov 02 22:19:50.993956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.993937 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:50.994035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.993987 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5sf\" (UniqueName: \"kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:50.994134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:50.994117 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.095468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.095432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5sf\" (UniqueName: \"kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.095659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.095603 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.095754 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.095731 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.096382 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.096362 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.097975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.097957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.156915 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.156847 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5sf\" (UniqueName: \"kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf\") pod \"maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.193307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.193275 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:51.574774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.574744 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8"] Nov 02 22:19:51.576384 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:19:51.576357 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22cf620_415a_4295_bcfd_cc9d9e739445.slice/crio-31c3680958e58a373ac6bbe82c0d47970216b6a865498f5152b165fcd78a8ec0 WatchSource:0}: Error finding container 31c3680958e58a373ac6bbe82c0d47970216b6a865498f5152b165fcd78a8ec0: Status 404 returned error can't find the container with id 31c3680958e58a373ac6bbe82c0d47970216b6a865498f5152b165fcd78a8ec0 Nov 02 22:19:51.717996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:51.717963 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" event={"ID":"d22cf620-415a-4295-bcfd-cc9d9e739445","Type":"ContainerStarted","Data":"31c3680958e58a373ac6bbe82c0d47970216b6a865498f5152b165fcd78a8ec0"} Nov 02 22:19:52.736033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:52.735935 5091 generic.go:356] "Generic (PLEG): container finished" podID="d22cf620-415a-4295-bcfd-cc9d9e739445" containerID="58a956f83430f794c49a98469307ae26a4f0ecd75b6e259c5a8b07bec57ed16a" exitCode=0 Nov 02 22:19:52.736033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:52.735997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" event={"ID":"d22cf620-415a-4295-bcfd-cc9d9e739445","Type":"ContainerDied","Data":"58a956f83430f794c49a98469307ae26a4f0ecd75b6e259c5a8b07bec57ed16a"} Nov 02 22:19:53.885090 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:53.885066 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:54.023962 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.023883 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf\" (UniqueName: \"kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf\") pod \"d22cf620-415a-4295-bcfd-cc9d9e739445\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " Nov 02 22:19:54.024129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.024042 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca\") pod \"d22cf620-415a-4295-bcfd-cc9d9e739445\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " Nov 02 22:19:54.024129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.024087 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca\") pod \"d22cf620-415a-4295-bcfd-cc9d9e739445\" (UID: \"d22cf620-415a-4295-bcfd-cc9d9e739445\") " Nov 02 22:19:54.024499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.024471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca" pod "d22cf620-415a-4295-bcfd-cc9d9e739445" (UID: "d22cf620-415a-4295-bcfd-cc9d9e739445"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:54.026352 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.026321 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf" pod "d22cf620-415a-4295-bcfd-cc9d9e739445" (UID: "d22cf620-415a-4295-bcfd-cc9d9e739445"). InnerVolumeSpecName "kube-api-access-nl5sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:54.026619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.026592 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca" pod "d22cf620-415a-4295-bcfd-cc9d9e739445" (UID: "d22cf620-415a-4295-bcfd-cc9d9e739445"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:54.125071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.125042 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5sf\" (UniqueName: \"kubernetes.io/projected/d22cf620-415a-4295-bcfd-cc9d9e739445-kube-api-access-nl5sf\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:54.125071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.125070 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:54.125250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.125080 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22cf620-415a-4295-bcfd-cc9d9e739445-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:54.753063 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.753030 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" Nov 02 22:19:54.753063 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.753043 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8" event={"ID":"d22cf620-415a-4295-bcfd-cc9d9e739445","Type":"ContainerDied","Data":"31c3680958e58a373ac6bbe82c0d47970216b6a865498f5152b165fcd78a8ec0"} Nov 02 22:19:54.753276 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:54.753114 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c3680958e58a373ac6bbe82c0d47970216b6a865498f5152b165fcd78a8ec0" Nov 02 22:19:56.749281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:56.749248 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7"] Nov 02 22:19:56.749730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:56.749479 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" podUID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" containerName="discovery" containerID="cri-o://d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9" gracePeriod=30 Nov 02 22:19:57.017544 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.017471 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 22:19:57.155353 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.155325 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.155548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.155403 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.155548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.155428 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.155548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.155458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.155774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.155609 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.155987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.155965 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.156135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.156012 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap\") pod \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\" (UID: \"d3c153c9-cf4b-4292-85ec-71fd80ac9cab\") " Nov 02 22:19:57.156949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.156915 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:57.158278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.158246 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:57.158398 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.158373 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:57.158784 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.158735 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "kube-api-access-5tn8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:57.160088 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.160066 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tn8x\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-kube-api-access-5tn8x\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.160181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.160094 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.160181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.160110 5091 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-dns-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.160181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.160124 5091 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-csr-ca-configmap\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.162814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.162784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:57.162814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.162791 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:57.162814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.162805 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig" pod "d3c153c9-cf4b-4292-85ec-71fd80ac9cab" (UID: "d3c153c9-cf4b-4292-85ec-71fd80ac9cab"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:57.261194 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.261163 5091 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-istio-kubeconfig\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.261194 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.261193 5091 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-cacerts\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.261194 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.261208 5091 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c153c9-cf4b-4292-85ec-71fd80ac9cab-local-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.773178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.773144 5091 generic.go:356] "Generic (PLEG): container finished" podID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" containerID="d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9" exitCode=0 Nov 02 22:19:57.773629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.773186 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" Nov 02 22:19:57.773629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.773221 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" event={"ID":"d3c153c9-cf4b-4292-85ec-71fd80ac9cab","Type":"ContainerDied","Data":"d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9"} Nov 02 22:19:57.773629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.773273 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7" event={"ID":"d3c153c9-cf4b-4292-85ec-71fd80ac9cab","Type":"ContainerDied","Data":"ca18f20db367422bc08ab380315c75b2989958e1ece676b03145ad8b8381f979"} Nov 02 22:19:57.773629 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.773296 5091 scope.go:122] "RemoveContainer" containerID="d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9" Nov 02 22:19:57.786033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.785990 5091 scope.go:122] "RemoveContainer" containerID="d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9" Nov 02 22:19:57.786273 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:19:57.786250 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9\": container with ID starting with d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9 not found: ID does not exist" containerID="d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9" Nov 02 22:19:57.786369 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.786279 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9"} err="failed to get container status \"d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9\": rpc error: code = NotFound desc = could not find container \"d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9\": container with ID starting with d8c7f36a4788f2e8fa6da1085a26f5ae2a3d10cd592ce048f0d2d1c65d9f84c9 not found: ID does not exist" Nov 02 22:19:57.993740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.993708 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7"] Nov 02 22:19:57.996952 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:57.996922 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hcgc7"] Nov 02 22:19:58.975951 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:19:58.975911 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" path="/var/lib/kubelet/pods/d3c153c9-cf4b-4292-85ec-71fd80ac9cab/volumes" Nov 02 22:20:07.112065 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:20:07.112029 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:20:18.497184 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:18.497149 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2"] Nov 02 22:20:18.497732 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:18.497405 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" containerID="cri-o://25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c" gracePeriod=30 Nov 02 22:20:19.050777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:19.050743 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw"] Nov 02 22:20:19.056023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:19.055997 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7ab6b5-6glcw"] Nov 02 22:20:20.886051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:20.886020 5091 scope.go:122] "RemoveContainer" containerID="75c92579e71f1a35b5fd423c2c90c461cbd48b9538ae253e99f37e98e80a54a5" Nov 02 22:20:20.895901 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:20.895879 5091 scope.go:122] "RemoveContainer" containerID="2c7e6f7b6c14fa8a0855d5fefd4e6af42f4f95ef15888acfc53d6f16cc51aa13" Nov 02 22:20:20.905435 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:20.905412 5091 scope.go:122] "RemoveContainer" containerID="c9e490aac68e29a0d68c77c6ed1993f825825a801e7fc59c04b5bfac5af2b502" Nov 02 22:20:20.975892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:20.975869 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0ca119-617d-4312-9f91-59faea5cc71f" path="/var/lib/kubelet/pods/ab0ca119-617d-4312-9f91-59faea5cc71f/volumes" Nov 02 22:20:23.747752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.747729 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 22:20:23.770736 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.770661 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8"] Nov 02 22:20:23.818534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.818741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818577 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.818741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.818741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818656 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.818741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818693 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.818741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818722 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818786 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818828 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818897 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs\") pod \"76f7f515-7919-4c2c-bb66-42a3054bc682\" (UID: \"76f7f515-7919-4c2c-bb66-42a3054bc682\") " Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.818925 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.819096 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.819317 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.819353 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.819357 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.819593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.819390 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.820861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.820832 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.821375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.821337 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:23.821571 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.821543 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.821801 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.821777 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:23.821910 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.821797 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af9e7c-x95v8"] Nov 02 22:20:23.822106 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.822085 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2" pod "76f7f515-7919-4c2c-bb66-42a3054bc682" (UID: "76f7f515-7919-4c2c-bb66-42a3054bc682"). InnerVolumeSpecName "kube-api-access-nrmh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:23.920278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920241 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.920278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920272 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f7f515-7919-4c2c-bb66-42a3054bc682-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.920493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920286 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.920493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920299 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.920493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920314 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrmh2\" (UniqueName: \"kubernetes.io/projected/76f7f515-7919-4c2c-bb66-42a3054bc682-kube-api-access-nrmh2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.920493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920327 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f7f515-7919-4c2c-bb66-42a3054bc682-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.920493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.920338 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f7f515-7919-4c2c-bb66-42a3054bc682-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.943130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.943099 5091 generic.go:356] "Generic (PLEG): container finished" podID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerID="25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c" exitCode=0 Nov 02 22:20:23.943294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.943146 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" Nov 02 22:20:23.943294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.943193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" event={"ID":"76f7f515-7919-4c2c-bb66-42a3054bc682","Type":"ContainerDied","Data":"25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c"} Nov 02 22:20:23.943294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.943256 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2" event={"ID":"76f7f515-7919-4c2c-bb66-42a3054bc682","Type":"ContainerDied","Data":"11399d8d89493c01cec4858db3175889e43baebfcb50f25cc06814a6151e56d0"} Nov 02 22:20:23.943294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.943285 5091 scope.go:122] "RemoveContainer" containerID="25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c" Nov 02 22:20:23.954506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.954489 5091 scope.go:122] "RemoveContainer" containerID="25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c" Nov 02 22:20:23.954797 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:20:23.954779 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c\": container with ID starting with 25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c not found: ID does not exist" containerID="25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c" Nov 02 22:20:23.954858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:23.954806 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c"} err="failed to get container status \"25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c\": rpc error: code = NotFound desc = could not find container \"25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c\": container with ID starting with 25e02a7075d7f4661d3bf34b15b7757e371480b58a1e36d3d908cc5b60b59d6c not found: ID does not exist" Nov 02 22:20:24.111176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:24.111091 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2"] Nov 02 22:20:24.117655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:24.117623 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-l79q2"] Nov 02 22:20:24.975818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:24.975785 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" path="/var/lib/kubelet/pods/76f7f515-7919-4c2c-bb66-42a3054bc682/volumes" Nov 02 22:20:24.976249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:24.976235 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22cf620-415a-4295-bcfd-cc9d9e739445" path="/var/lib/kubelet/pods/d22cf620-415a-4295-bcfd-cc9d9e739445/volumes" Nov 02 22:20:31.394495 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394465 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr"] Nov 02 22:20:31.395014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394916 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d22cf620-415a-4295-bcfd-cc9d9e739445" containerName="revoke-keys" Nov 02 22:20:31.395014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394931 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22cf620-415a-4295-bcfd-cc9d9e739445" containerName="revoke-keys" Nov 02 22:20:31.395014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394967 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" containerName="discovery" Nov 02 22:20:31.395014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394973 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" containerName="discovery" Nov 02 22:20:31.395014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394983 5091 cpu_manager.go:414] "containerMap: removing container" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" Nov 02 22:20:31.395014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.394988 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" Nov 02 22:20:31.395354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.395081 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c153c9-cf4b-4292-85ec-71fd80ac9cab" containerName="discovery" Nov 02 22:20:31.395354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.395093 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f7f515-7919-4c2c-bb66-42a3054bc682" containerName="istio-proxy" Nov 02 22:20:31.395354 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.395102 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22cf620-415a-4295-bcfd-cc9d9e739445" containerName="revoke-keys" Nov 02 22:20:31.399807 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.399784 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.452486 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.452458 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:20:31.452777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.452757 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:20:31.495177 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.495147 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.495177 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.495180 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqnr\" (UniqueName: \"kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.495382 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.495340 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.502340 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.502319 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:31.502527 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.502507 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:20:31.504977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.504951 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr"] Nov 02 22:20:31.596879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.596837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.597061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.596906 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.597061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.596925 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqnr\" (UniqueName: \"kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.597516 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.597489 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.599251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.599222 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.656464 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.656390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqnr\" (UniqueName: \"kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr\") pod \"maas-api-revoke-sub-20b26a0d-rfcrr\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:31.713136 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:31.713109 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:32.103512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:32.103480 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr"] Nov 02 22:20:32.105626 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:20:32.105600 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3187f782_a32a_4eb8_8165_8e331536d770.slice/crio-cf9b4d9309caeb8b6245946271a280d168e86292a6cfe452a3e89734029406fe WatchSource:0}: Error finding container cf9b4d9309caeb8b6245946271a280d168e86292a6cfe452a3e89734029406fe: Status 404 returned error can't find the container with id cf9b4d9309caeb8b6245946271a280d168e86292a6cfe452a3e89734029406fe Nov 02 22:20:32.991660 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:32.991623 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb"] Nov 02 22:20:32.995410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:32.995388 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.002333 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.002303 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" event={"ID":"3187f782-a32a-4eb8-8165-8e331536d770","Type":"ContainerStarted","Data":"2382ebdd8597678f1aaa3a61bb8a6b1310e011d4e6606d68c27f4cc7473700d2"} Nov 02 22:20:33.002437 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.002351 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" event={"ID":"3187f782-a32a-4eb8-8165-8e331536d770","Type":"ContainerStarted","Data":"cf9b4d9309caeb8b6245946271a280d168e86292a6cfe452a3e89734029406fe"} Nov 02 22:20:33.069827 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.069801 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb"] Nov 02 22:20:33.114386 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.114350 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.114644 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.114619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55lj\" (UniqueName: \"kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.114764 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.114741 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.215508 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.215475 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.215712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.215561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55lj\" (UniqueName: \"kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.215712 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.215625 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.216113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.216095 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.217978 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.217950 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.242578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.242504 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" podStartSLOduration=2.242492195 podStartE2EDuration="2.242492195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:33.240323855 +0000 UTC m=+2780.402418975" watchObservedRunningTime="2026-11-02 22:20:33.242492195 +0000 UTC m=+2780.404587314" Nov 02 22:20:33.279548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.279518 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55lj\" (UniqueName: \"kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.309808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.309784 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:33.688943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:33.688908 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb"] Nov 02 22:20:33.689795 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:20:33.689737 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba89aa5_e0c7_4c48_a541_9ae8b4956144.slice/crio-389ce90d3a10c8ec6a20c08bb7be6e66604c13b076119a8b1ebfcfbeb6917132 WatchSource:0}: Error finding container 389ce90d3a10c8ec6a20c08bb7be6e66604c13b076119a8b1ebfcfbeb6917132: Status 404 returned error can't find the container with id 389ce90d3a10c8ec6a20c08bb7be6e66604c13b076119a8b1ebfcfbeb6917132 Nov 02 22:20:34.009523 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:34.009487 5091 generic.go:356] "Generic (PLEG): container finished" podID="3187f782-a32a-4eb8-8165-8e331536d770" containerID="2382ebdd8597678f1aaa3a61bb8a6b1310e011d4e6606d68c27f4cc7473700d2" exitCode=0 Nov 02 22:20:34.009947 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:34.009574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" event={"ID":"3187f782-a32a-4eb8-8165-8e331536d770","Type":"ContainerDied","Data":"2382ebdd8597678f1aaa3a61bb8a6b1310e011d4e6606d68c27f4cc7473700d2"} Nov 02 22:20:34.011275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:34.011249 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" event={"ID":"aba89aa5-e0c7-4c48-a541-9ae8b4956144","Type":"ContainerStarted","Data":"009dc33305d3b97d74192067d558462ddd6bda8aa4e164d1e3d91facfa754ace"} Nov 02 22:20:34.011375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:34.011299 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" event={"ID":"aba89aa5-e0c7-4c48-a541-9ae8b4956144","Type":"ContainerStarted","Data":"389ce90d3a10c8ec6a20c08bb7be6e66604c13b076119a8b1ebfcfbeb6917132"} Nov 02 22:20:34.248992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:34.248891 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" podStartSLOduration=2.248878014 podStartE2EDuration="2.248878014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:34.24684559 +0000 UTC m=+2781.408940747" watchObservedRunningTime="2026-11-02 22:20:34.248878014 +0000 UTC m=+2781.410973134" Nov 02 22:20:35.018680 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.018650 5091 generic.go:356] "Generic (PLEG): container finished" podID="aba89aa5-e0c7-4c48-a541-9ae8b4956144" containerID="009dc33305d3b97d74192067d558462ddd6bda8aa4e164d1e3d91facfa754ace" exitCode=0 Nov 02 22:20:35.019071 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.018739 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" event={"ID":"aba89aa5-e0c7-4c48-a541-9ae8b4956144","Type":"ContainerDied","Data":"009dc33305d3b97d74192067d558462ddd6bda8aa4e164d1e3d91facfa754ace"} Nov 02 22:20:35.145804 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.145778 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:35.238269 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.238238 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr\" (UniqueName: \"kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr\") pod \"3187f782-a32a-4eb8-8165-8e331536d770\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " Nov 02 22:20:35.238467 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.238324 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca\") pod \"3187f782-a32a-4eb8-8165-8e331536d770\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " Nov 02 22:20:35.238467 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.238386 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca\") pod \"3187f782-a32a-4eb8-8165-8e331536d770\" (UID: \"3187f782-a32a-4eb8-8165-8e331536d770\") " Nov 02 22:20:35.238809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.238778 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca" pod "3187f782-a32a-4eb8-8165-8e331536d770" (UID: "3187f782-a32a-4eb8-8165-8e331536d770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:35.240391 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.240371 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca" pod "3187f782-a32a-4eb8-8165-8e331536d770" (UID: "3187f782-a32a-4eb8-8165-8e331536d770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:35.240498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.240455 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr" pod "3187f782-a32a-4eb8-8165-8e331536d770" (UID: "3187f782-a32a-4eb8-8165-8e331536d770"). InnerVolumeSpecName "kube-api-access-7xqnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:35.339877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.339809 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xqnr\" (UniqueName: \"kubernetes.io/projected/3187f782-a32a-4eb8-8165-8e331536d770-kube-api-access-7xqnr\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:35.339877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.339834 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:35.339877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:35.339844 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3187f782-a32a-4eb8-8165-8e331536d770-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:36.025870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.025837 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" Nov 02 22:20:36.025870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.025852 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr" event={"ID":"3187f782-a32a-4eb8-8165-8e331536d770","Type":"ContainerDied","Data":"cf9b4d9309caeb8b6245946271a280d168e86292a6cfe452a3e89734029406fe"} Nov 02 22:20:36.026388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.025910 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf9b4d9309caeb8b6245946271a280d168e86292a6cfe452a3e89734029406fe" Nov 02 22:20:36.174949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.174923 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:36.247855 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.247826 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca\") pod \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " Nov 02 22:20:36.248028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.247970 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj\" (UniqueName: \"kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj\") pod \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " Nov 02 22:20:36.248028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.248007 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca\") pod \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\" (UID: \"aba89aa5-e0c7-4c48-a541-9ae8b4956144\") " Nov 02 22:20:36.248415 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.248391 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca" pod "aba89aa5-e0c7-4c48-a541-9ae8b4956144" (UID: "aba89aa5-e0c7-4c48-a541-9ae8b4956144"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:36.249844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.249824 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca" pod "aba89aa5-e0c7-4c48-a541-9ae8b4956144" (UID: "aba89aa5-e0c7-4c48-a541-9ae8b4956144"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:36.250176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.250156 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj" pod "aba89aa5-e0c7-4c48-a541-9ae8b4956144" (UID: "aba89aa5-e0c7-4c48-a541-9ae8b4956144"). InnerVolumeSpecName "kube-api-access-f55lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:36.349410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.349334 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:36.349410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.349365 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55lj\" (UniqueName: \"kubernetes.io/projected/aba89aa5-e0c7-4c48-a541-9ae8b4956144-kube-api-access-f55lj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:36.349410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:36.349376 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba89aa5-e0c7-4c48-a541-9ae8b4956144-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:37.033924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:37.033896 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" Nov 02 22:20:37.034323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:37.033924 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb" event={"ID":"aba89aa5-e0c7-4c48-a541-9ae8b4956144","Type":"ContainerDied","Data":"389ce90d3a10c8ec6a20c08bb7be6e66604c13b076119a8b1ebfcfbeb6917132"} Nov 02 22:20:37.034323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:37.033978 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="389ce90d3a10c8ec6a20c08bb7be6e66604c13b076119a8b1ebfcfbeb6917132" Nov 02 22:20:37.112642 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:20:37.112609 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:20:44.340630 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.340596 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9"] Nov 02 22:20:44.341052 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.341038 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3187f782-a32a-4eb8-8165-8e331536d770" containerName="revoke-keys" Nov 02 22:20:44.341093 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.341052 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3187f782-a32a-4eb8-8165-8e331536d770" containerName="revoke-keys" Nov 02 22:20:44.341154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.341094 5091 cpu_manager.go:414] "containerMap: removing container" podUID="aba89aa5-e0c7-4c48-a541-9ae8b4956144" containerName="revoke-keys" Nov 02 22:20:44.341154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.341099 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba89aa5-e0c7-4c48-a541-9ae8b4956144" containerName="revoke-keys" Nov 02 22:20:44.341225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.341167 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3187f782-a32a-4eb8-8165-8e331536d770" containerName="revoke-keys" Nov 02 22:20:44.341225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.341182 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba89aa5-e0c7-4c48-a541-9ae8b4956144" containerName="revoke-keys" Nov 02 22:20:44.344380 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.344359 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.403102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.403074 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9"] Nov 02 22:20:44.429270 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.429241 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.429389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.429349 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.429443 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.429424 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl9z\" (UniqueName: \"kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.449280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.449247 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:20:44.449280 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.449247 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:20:44.449450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.449258 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:20:44.449450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.449305 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:20:44.530380 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.530350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.530522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.530408 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl9z\" (UniqueName: \"kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.530522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.530455 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.531113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.531091 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.532884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.532863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.587210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.587186 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl9z\" (UniqueName: \"kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z\") pod \"maas-api-revoke-sub-52049a7f-lwfg9\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:44.656865 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:44.656840 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:45.031298 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:45.031272 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9"] Nov 02 22:20:45.032845 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:20:45.032819 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc230e278_7505_4fde_a5f5_e53561045821.slice/crio-ebd5df187cc656c8fcbe37923bce586454c0c0480cd6f5a6f798a7d1306f5901 WatchSource:0}: Error finding container ebd5df187cc656c8fcbe37923bce586454c0c0480cd6f5a6f798a7d1306f5901: Status 404 returned error can't find the container with id ebd5df187cc656c8fcbe37923bce586454c0c0480cd6f5a6f798a7d1306f5901 Nov 02 22:20:45.084924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:45.084892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" event={"ID":"c230e278-7505-4fde-a5f5-e53561045821","Type":"ContainerStarted","Data":"ebd5df187cc656c8fcbe37923bce586454c0c0480cd6f5a6f798a7d1306f5901"} Nov 02 22:20:46.093644 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:46.093612 5091 generic.go:356] "Generic (PLEG): container finished" podID="c230e278-7505-4fde-a5f5-e53561045821" containerID="9e36239df65a9bf2ea7855a87dd3d207f642869f8c42636586c6892623f1961b" exitCode=0 Nov 02 22:20:46.094033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:46.093698 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" event={"ID":"c230e278-7505-4fde-a5f5-e53561045821","Type":"ContainerDied","Data":"9e36239df65a9bf2ea7855a87dd3d207f642869f8c42636586c6892623f1961b"} Nov 02 22:20:47.247687 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.247652 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:47.359309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.359211 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca\") pod \"c230e278-7505-4fde-a5f5-e53561045821\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " Nov 02 22:20:47.359489 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.359362 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca\") pod \"c230e278-7505-4fde-a5f5-e53561045821\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " Nov 02 22:20:47.359489 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.359425 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z\" (UniqueName: \"kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z\") pod \"c230e278-7505-4fde-a5f5-e53561045821\" (UID: \"c230e278-7505-4fde-a5f5-e53561045821\") " Nov 02 22:20:47.359634 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.359596 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca" pod "c230e278-7505-4fde-a5f5-e53561045821" (UID: "c230e278-7505-4fde-a5f5-e53561045821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:47.359848 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.359830 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c230e278-7505-4fde-a5f5-e53561045821-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:47.361706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.361647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z" pod "c230e278-7505-4fde-a5f5-e53561045821" (UID: "c230e278-7505-4fde-a5f5-e53561045821"). InnerVolumeSpecName "kube-api-access-wnl9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:47.361885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.361863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca" pod "c230e278-7505-4fde-a5f5-e53561045821" (UID: "c230e278-7505-4fde-a5f5-e53561045821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:47.461189 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.461154 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c230e278-7505-4fde-a5f5-e53561045821-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:47.461189 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:47.461185 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnl9z\" (UniqueName: \"kubernetes.io/projected/c230e278-7505-4fde-a5f5-e53561045821-kube-api-access-wnl9z\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:48.106618 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:48.106587 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" Nov 02 22:20:48.106618 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:48.106605 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9" event={"ID":"c230e278-7505-4fde-a5f5-e53561045821","Type":"ContainerDied","Data":"ebd5df187cc656c8fcbe37923bce586454c0c0480cd6f5a6f798a7d1306f5901"} Nov 02 22:20:48.106857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:20:48.106651 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd5df187cc656c8fcbe37923bce586454c0c0480cd6f5a6f798a7d1306f5901" Nov 02 22:21:03.733734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:03.733693 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw"] Nov 02 22:21:03.734350 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:03.733994 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" containerID="cri-o://cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded" gracePeriod=30 Nov 02 22:21:07.123337 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:21:07.123289 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:21:08.989481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:08.989454 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:21:09.011092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.011028 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb"] Nov 02 22:21:09.058954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.058926 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.058967 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.058988 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059021 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059126 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059160 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059195 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059232 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb0367-56qrb"] Nov 02 22:21:09.059296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059238 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:09.059575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059332 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:09.059575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059353 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo\") pod \"2698ccdd-d9bc-4566-8665-56839a2e6e86\" (UID: \"2698ccdd-d9bc-4566-8665-56839a2e6e86\") " Nov 02 22:21:09.059575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059478 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:09.059813 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059736 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:09.059877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059853 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.059877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059873 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698ccdd-d9bc-4566-8665-56839a2e6e86-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.059960 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059887 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.059960 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059902 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.060027 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.059971 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:09.061255 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.061235 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:09.061406 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.061391 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "kube-api-access-bdvrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:09.061713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.061687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:21:09.061818 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.061716 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy" pod "2698ccdd-d9bc-4566-8665-56839a2e6e86" (UID: "2698ccdd-d9bc-4566-8665-56839a2e6e86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:09.160548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.160513 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.160548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.160542 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvrc\" (UniqueName: \"kubernetes.io/projected/2698ccdd-d9bc-4566-8665-56839a2e6e86-kube-api-access-bdvrc\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.160548 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.160553 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.160805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.160562 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.160805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.160572 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698ccdd-d9bc-4566-8665-56839a2e6e86-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:09.245752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.245717 5091 generic.go:356] "Generic (PLEG): container finished" podID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerID="cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded" exitCode=0 Nov 02 22:21:09.245924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.245769 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" Nov 02 22:21:09.245924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.245793 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" event={"ID":"2698ccdd-d9bc-4566-8665-56839a2e6e86","Type":"ContainerDied","Data":"cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded"} Nov 02 22:21:09.245924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.245851 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw" event={"ID":"2698ccdd-d9bc-4566-8665-56839a2e6e86","Type":"ContainerDied","Data":"9f5f0a0a71a6a77a148c4769b24065be3178f0bd61d25e72828636f76c4dc060"} Nov 02 22:21:09.245924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.245873 5091 scope.go:122] "RemoveContainer" containerID="cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded" Nov 02 22:21:09.256592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.256574 5091 scope.go:122] "RemoveContainer" containerID="cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded" Nov 02 22:21:09.256895 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:21:09.256876 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded\": container with ID starting with cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded not found: ID does not exist" containerID="cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded" Nov 02 22:21:09.256982 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.256903 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded"} err="failed to get container status \"cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded\": rpc error: code = NotFound desc = could not find container \"cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded\": container with ID starting with cd73f8a8b55016124e11421f99cd4f11305589416202092319a1ea7a3fd89ded not found: ID does not exist" Nov 02 22:21:09.409576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.409542 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw"] Nov 02 22:21:09.415822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:09.415791 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-974qw"] Nov 02 22:21:10.975954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:10.975922 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" path="/var/lib/kubelet/pods/2698ccdd-d9bc-4566-8665-56839a2e6e86/volumes" Nov 02 22:21:10.976370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:10.976356 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba89aa5-e0c7-4c48-a541-9ae8b4956144" path="/var/lib/kubelet/pods/aba89aa5-e0c7-4c48-a541-9ae8b4956144/volumes" Nov 02 22:21:15.643094 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:15.643064 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj"] Nov 02 22:21:15.643534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:15.643263 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" containerID="cri-o://8d6e0837a8b31cb2a816655e3da584280a8b97141edc32c47770d165592c1b57" gracePeriod=30 Nov 02 22:21:16.518912 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.518871 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6"] Nov 02 22:21:16.519388 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.519369 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c230e278-7505-4fde-a5f5-e53561045821" containerName="revoke-keys" Nov 02 22:21:16.519500 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.519389 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c230e278-7505-4fde-a5f5-e53561045821" containerName="revoke-keys" Nov 02 22:21:16.519500 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.519408 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" Nov 02 22:21:16.519500 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.519415 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" Nov 02 22:21:16.519705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.519574 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c230e278-7505-4fde-a5f5-e53561045821" containerName="revoke-keys" Nov 02 22:21:16.519705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.519588 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2698ccdd-d9bc-4566-8665-56839a2e6e86" containerName="istio-proxy" Nov 02 22:21:16.522844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.522824 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.576079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.576052 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:16.577342 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.577318 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:21:16.577565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.577543 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:21:16.577720 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.577614 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:21:16.580491 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.580469 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6"] Nov 02 22:21:16.632895 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.632868 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kqk\" (UniqueName: \"kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.633028 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.632902 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.633079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.633036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.733565 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.733534 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.734070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.733626 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kqk\" (UniqueName: \"kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.734070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.733661 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.734410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.734386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.735916 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.735893 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.789826 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.789765 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kqk\" (UniqueName: \"kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:16.835767 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:16.835745 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:17.209651 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:17.209624 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6"] Nov 02 22:21:17.210956 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:21:17.210927 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddc8a27_4e5f_4f17_987b_3dc8bc750c8c.slice/crio-cd94a1e8c80fd69222cc9e9a5b08bdd352cc786063e4c9f70e961f0ba084a74b WatchSource:0}: Error finding container cd94a1e8c80fd69222cc9e9a5b08bdd352cc786063e4c9f70e961f0ba084a74b: Status 404 returned error can't find the container with id cd94a1e8c80fd69222cc9e9a5b08bdd352cc786063e4c9f70e961f0ba084a74b Nov 02 22:21:17.302982 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:17.302947 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" event={"ID":"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c","Type":"ContainerStarted","Data":"cd94a1e8c80fd69222cc9e9a5b08bdd352cc786063e4c9f70e961f0ba084a74b"} Nov 02 22:21:18.311286 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:18.311201 5091 generic.go:356] "Generic (PLEG): container finished" podID="1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" containerID="637a1f72b9ab1e5cd838a8d7ccf4f5489915462f4164060fdea33d04f4305f03" exitCode=0 Nov 02 22:21:18.311706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:18.311284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" event={"ID":"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c","Type":"ContainerDied","Data":"637a1f72b9ab1e5cd838a8d7ccf4f5489915462f4164060fdea33d04f4305f03"} Nov 02 22:21:19.201225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.201190 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz"] Nov 02 22:21:19.455426 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.455372 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:19.561522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.561482 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca\") pod \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " Nov 02 22:21:19.561718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.561571 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca\") pod \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " Nov 02 22:21:19.561718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.561634 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk\" (UniqueName: \"kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk\") pod \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\" (UID: \"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c\") " Nov 02 22:21:19.561974 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.561948 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca" pod "1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" (UID: "1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:19.563708 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.563688 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk" pod "1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" (UID: "1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c"). InnerVolumeSpecName "kube-api-access-m9kqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:19.563815 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.563796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca" pod "1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" (UID: "1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:19.662844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.662805 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9kqk\" (UniqueName: \"kubernetes.io/projected/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-kube-api-access-m9kqk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:19.662844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.662838 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:19.663061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.662854 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:19.669160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.669130 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw"] Nov 02 22:21:19.669585 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.669572 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" containerName="revoke-keys" Nov 02 22:21:19.669641 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.669586 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" containerName="revoke-keys" Nov 02 22:21:19.669738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.669728 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" containerName="revoke-keys" Nov 02 22:21:19.673008 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.672989 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.730264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.730174 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw"] Nov 02 22:21:19.763776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.763744 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rkr\" (UniqueName: \"kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.763945 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.763786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.763990 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.763949 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.864715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.864659 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.864883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.864775 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rkr\" (UniqueName: \"kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.864883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.864799 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.865334 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.865313 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.867239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.867220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.920922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.920891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rkr\" (UniqueName: \"kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr\") pod \"maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:19.986968 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:19.986905 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:20.325084 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.325011 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" Nov 02 22:21:20.325084 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.325048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6" event={"ID":"1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c","Type":"ContainerDied","Data":"cd94a1e8c80fd69222cc9e9a5b08bdd352cc786063e4c9f70e961f0ba084a74b"} Nov 02 22:21:20.325268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.325095 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd94a1e8c80fd69222cc9e9a5b08bdd352cc786063e4c9f70e961f0ba084a74b" Nov 02 22:21:20.361475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.361449 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw"] Nov 02 22:21:20.362554 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:21:20.362526 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad40ed7e_bd8d_427c_a55c_243976448330.slice/crio-6b43b455e3782d70b86e6fca2eac2c63dc88bc8996e0e4a92c23dbc9171693b0 WatchSource:0}: Error finding container 6b43b455e3782d70b86e6fca2eac2c63dc88bc8996e0e4a92c23dbc9171693b0: Status 404 returned error can't find the container with id 6b43b455e3782d70b86e6fca2eac2c63dc88bc8996e0e4a92c23dbc9171693b0 Nov 02 22:21:20.897638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.897616 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:21:20.951209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.951175 5091 scope.go:122] "RemoveContainer" containerID="94d313b83d2d36b5306bfd87171ea085de9b86a12c1c05d05d8dd916c6f3c0d8" Nov 02 22:21:20.962529 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.962505 5091 scope.go:122] "RemoveContainer" containerID="24f3e322bcdde21764a23d660f25f7d2ae930dbd91a7c0d84b2b236e08739923" Nov 02 22:21:20.972294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.972267 5091 scope.go:122] "RemoveContainer" containerID="8d6e0837a8b31cb2a816655e3da584280a8b97141edc32c47770d165592c1b57" Nov 02 22:21:20.977306 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977331 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977379 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977408 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977437 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977484 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977532 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:20.977832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977643 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977665 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:20.977832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977687 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.977832 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.977786 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert\") pod \"11eed396-dc54-4649-8579-ae882d4bbac3\" (UID: \"11eed396-dc54-4649-8579-ae882d4bbac3\") " Nov 02 22:21:20.978216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.978159 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:20.978216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.978185 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:20.978361 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.978318 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:20.979655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.979619 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:20.980093 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.980074 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:21:20.980175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.980091 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "kube-api-access-8f96c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:20.980776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.980748 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:20.980869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.980775 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:20.980869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.980790 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data" pod "11eed396-dc54-4649-8579-ae882d4bbac3" (UID: "11eed396-dc54-4649-8579-ae882d4bbac3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:20.982300 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.982283 5091 scope.go:122] "RemoveContainer" containerID="f237500afb3bdca6c836adbd8d50d8a57be9b72aca82d94595e507701f90cac4" Nov 02 22:21:20.995251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:20.995229 5091 scope.go:122] "RemoveContainer" containerID="cf21465d4acbcc314a166e5fcbf1fcff70e9843a949e75b12f7719157289c2e7" Nov 02 22:21:21.003207 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.003188 5091 scope.go:122] "RemoveContainer" containerID="5f3a997808f1e011908d784edd008421c48fad74ac4516c50b5a4c02179bb805" Nov 02 22:21:21.011701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.011655 5091 scope.go:122] "RemoveContainer" containerID="7b40bbc0312f0cde51259457dfef3a80ca781a92c7a9ac3c2df588ee13f2c210" Nov 02 22:21:21.021550 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.021523 5091 scope.go:122] "RemoveContainer" containerID="4990c26dcdeef26d618bf2d8866c2f1dd5a342bd4b1adb54bc632b68c25f2221" Nov 02 22:21:21.033954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.033931 5091 scope.go:122] "RemoveContainer" containerID="9e0be7c1beff3a4118ccc9d023b36f2726235875715a127b909461ce173670be" Nov 02 22:21:21.079043 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079019 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f96c\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-kube-api-access-8f96c\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.079130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079046 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.079130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079058 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eed396-dc54-4649-8579-ae882d4bbac3-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.079130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079067 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.079130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079076 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eed396-dc54-4649-8579-ae882d4bbac3-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.079130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079086 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eed396-dc54-4649-8579-ae882d4bbac3-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.079130 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.079094 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eed396-dc54-4649-8579-ae882d4bbac3-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:21.332973 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.332936 5091 generic.go:356] "Generic (PLEG): container finished" podID="ad40ed7e-bd8d-427c-a55c-243976448330" containerID="c6ce02146c3aac076f5b8cb30f563d2f4edaf1701fc2287de48647e63c220c95" exitCode=0 Nov 02 22:21:21.333155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.332975 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" event={"ID":"ad40ed7e-bd8d-427c-a55c-243976448330","Type":"ContainerDied","Data":"c6ce02146c3aac076f5b8cb30f563d2f4edaf1701fc2287de48647e63c220c95"} Nov 02 22:21:21.333155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.333033 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" event={"ID":"ad40ed7e-bd8d-427c-a55c-243976448330","Type":"ContainerStarted","Data":"6b43b455e3782d70b86e6fca2eac2c63dc88bc8996e0e4a92c23dbc9171693b0"} Nov 02 22:21:21.333155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.333052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" event={"ID":"11eed396-dc54-4649-8579-ae882d4bbac3","Type":"ContainerDied","Data":"8d6e0837a8b31cb2a816655e3da584280a8b97141edc32c47770d165592c1b57"} Nov 02 22:21:21.333155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.333071 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" event={"ID":"11eed396-dc54-4649-8579-ae882d4bbac3","Type":"ContainerDied","Data":"5b67d1d419da5ea74c136fcc2cf0f596710e51af162ced925cc572bf9353efe3"} Nov 02 22:21:21.333155 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.333119 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj" Nov 02 22:21:21.619332 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.619302 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj"] Nov 02 22:21:21.623397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:21.623369 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb0367-openshift-default-99b5ddbbc-jnscj"] Nov 02 22:21:22.472985 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.472958 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:22.594421 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.594359 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca\") pod \"ad40ed7e-bd8d-427c-a55c-243976448330\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " Nov 02 22:21:22.594574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.594432 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca\") pod \"ad40ed7e-bd8d-427c-a55c-243976448330\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " Nov 02 22:21:22.594574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.594545 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr\" (UniqueName: \"kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr\") pod \"ad40ed7e-bd8d-427c-a55c-243976448330\" (UID: \"ad40ed7e-bd8d-427c-a55c-243976448330\") " Nov 02 22:21:22.594824 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.594798 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca" pod "ad40ed7e-bd8d-427c-a55c-243976448330" (UID: "ad40ed7e-bd8d-427c-a55c-243976448330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:22.596605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.596573 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr" pod "ad40ed7e-bd8d-427c-a55c-243976448330" (UID: "ad40ed7e-bd8d-427c-a55c-243976448330"). InnerVolumeSpecName "kube-api-access-46rkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:22.596605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.596588 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca" pod "ad40ed7e-bd8d-427c-a55c-243976448330" (UID: "ad40ed7e-bd8d-427c-a55c-243976448330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:22.695364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.695330 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:22.695364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.695363 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46rkr\" (UniqueName: \"kubernetes.io/projected/ad40ed7e-bd8d-427c-a55c-243976448330-kube-api-access-46rkr\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:22.695576 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.695378 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad40ed7e-bd8d-427c-a55c-243976448330-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:22.975961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:22.975927 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" path="/var/lib/kubelet/pods/11eed396-dc54-4649-8579-ae882d4bbac3/volumes" Nov 02 22:21:23.347711 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:23.347599 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" event={"ID":"ad40ed7e-bd8d-427c-a55c-243976448330","Type":"ContainerDied","Data":"6b43b455e3782d70b86e6fca2eac2c63dc88bc8996e0e4a92c23dbc9171693b0"} Nov 02 22:21:23.347711 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:23.347619 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw" Nov 02 22:21:23.347711 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:23.347650 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b43b455e3782d70b86e6fca2eac2c63dc88bc8996e0e4a92c23dbc9171693b0" Nov 02 22:21:34.221428 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.221308 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" podUID="2add07da-f806-4412-89cb-43aa2a40f3d4" containerName="main" containerID="cri-o://69903c391051a436c1dccf972f891a14b397ab270740f080e614876cfe09ef8a" gracePeriod=45 Nov 02 22:21:34.414819 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.414782 5091 generic.go:356] "Generic (PLEG): container finished" podID="2add07da-f806-4412-89cb-43aa2a40f3d4" containerID="69903c391051a436c1dccf972f891a14b397ab270740f080e614876cfe09ef8a" exitCode=0 Nov 02 22:21:34.414995 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.414876 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" event={"ID":"2add07da-f806-4412-89cb-43aa2a40f3d4","Type":"ContainerDied","Data":"69903c391051a436c1dccf972f891a14b397ab270740f080e614876cfe09ef8a"} Nov 02 22:21:34.473037 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.472979 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:21:34.605792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.605758 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4\" (UniqueName: \"kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4\") pod \"2add07da-f806-4412-89cb-43aa2a40f3d4\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " Nov 02 22:21:34.606105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.605810 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache\") pod \"2add07da-f806-4412-89cb-43aa2a40f3d4\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " Nov 02 22:21:34.606105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.605871 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs\") pod \"2add07da-f806-4412-89cb-43aa2a40f3d4\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " Nov 02 22:21:34.606105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.605903 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir\") pod \"2add07da-f806-4412-89cb-43aa2a40f3d4\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " Nov 02 22:21:34.606105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.605935 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home\") pod \"2add07da-f806-4412-89cb-43aa2a40f3d4\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " Nov 02 22:21:34.606105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.605961 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm\") pod \"2add07da-f806-4412-89cb-43aa2a40f3d4\" (UID: \"2add07da-f806-4412-89cb-43aa2a40f3d4\") " Nov 02 22:21:34.607100 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.607051 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home" pod "2add07da-f806-4412-89cb-43aa2a40f3d4" (UID: "2add07da-f806-4412-89cb-43aa2a40f3d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:34.607375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.607289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir" pod "2add07da-f806-4412-89cb-43aa2a40f3d4" (UID: "2add07da-f806-4412-89cb-43aa2a40f3d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:34.607375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.607355 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache" pod "2add07da-f806-4412-89cb-43aa2a40f3d4" (UID: "2add07da-f806-4412-89cb-43aa2a40f3d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:34.609721 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.609695 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs" pod "2add07da-f806-4412-89cb-43aa2a40f3d4" (UID: "2add07da-f806-4412-89cb-43aa2a40f3d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:34.609942 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.609917 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm" pod "2add07da-f806-4412-89cb-43aa2a40f3d4" (UID: "2add07da-f806-4412-89cb-43aa2a40f3d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:34.615353 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.615324 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4" pod "2add07da-f806-4412-89cb-43aa2a40f3d4" (UID: "2add07da-f806-4412-89cb-43aa2a40f3d4"). InnerVolumeSpecName "kube-api-access-h6dp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:34.707718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.707651 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6dp4\" (UniqueName: \"kubernetes.io/projected/2add07da-f806-4412-89cb-43aa2a40f3d4-kube-api-access-h6dp4\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.707718 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.707717 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.707899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.707730 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add07da-f806-4412-89cb-43aa2a40f3d4-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.707899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.707756 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.707899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.707764 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:34.707899 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:34.707772 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add07da-f806-4412-89cb-43aa2a40f3d4-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:35.422497 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:35.422461 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" event={"ID":"2add07da-f806-4412-89cb-43aa2a40f3d4","Type":"ContainerDied","Data":"b7f7b3e50e99e072b86d106d03434f7670882442ab2482becc6952e96caf505f"} Nov 02 22:21:35.423001 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:35.422514 5091 scope.go:122] "RemoveContainer" containerID="69903c391051a436c1dccf972f891a14b397ab270740f080e614876cfe09ef8a" Nov 02 22:21:35.423001 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:35.422470 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz" Nov 02 22:21:35.585385 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:35.585349 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz"] Nov 02 22:21:35.634976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:35.634945 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-94a8a3/test-model-94a8a3-kserve-5856784559-fnhmz"] Nov 02 22:21:36.975330 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:36.975296 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2add07da-f806-4412-89cb-43aa2a40f3d4" path="/var/lib/kubelet/pods/2add07da-f806-4412-89cb-43aa2a40f3d4/volumes" Nov 02 22:21:37.111812 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:21:37.111778 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:21:37.748298 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:37.748265 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw"] Nov 02 22:21:37.797507 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:37.797468 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d1a155-dbzsw"] Nov 02 22:21:38.976031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:38.975997 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad40ed7e-bd8d-427c-a55c-243976448330" path="/var/lib/kubelet/pods/ad40ed7e-bd8d-427c-a55c-243976448330/volumes" Nov 02 22:21:44.899014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.898971 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm"] Nov 02 22:21:44.899642 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899621 5091 cpu_manager.go:414] "containerMap: removing container" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" Nov 02 22:21:44.899747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899646 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" Nov 02 22:21:44.899747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899663 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ad40ed7e-bd8d-427c-a55c-243976448330" containerName="revoke-keys" Nov 02 22:21:44.899747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899688 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad40ed7e-bd8d-427c-a55c-243976448330" containerName="revoke-keys" Nov 02 22:21:44.899747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899741 5091 cpu_manager.go:414] "containerMap: removing container" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899751 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899769 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2add07da-f806-4412-89cb-43aa2a40f3d4" containerName="main" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899776 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2add07da-f806-4412-89cb-43aa2a40f3d4" containerName="main" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899888 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899900 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad40ed7e-bd8d-427c-a55c-243976448330" containerName="revoke-keys" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899919 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="11eed396-dc54-4649-8579-ae882d4bbac3" containerName="istio-proxy" Nov 02 22:21:44.899961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.899930 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2add07da-f806-4412-89cb-43aa2a40f3d4" containerName="main" Nov 02 22:21:44.903737 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.903711 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:44.957642 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:44.957608 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm"] Nov 02 22:21:45.004181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.004150 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:21:45.004325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.004187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.004325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.004251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.004325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.004315 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkxj\" (UniqueName: \"kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.006737 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.006720 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:45.006802 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.006751 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:21:45.006802 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.006792 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:21:45.105929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.105892 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.106240 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.106219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.106334 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.106258 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkxj\" (UniqueName: \"kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.106903 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.106877 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.108774 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.108755 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.162580 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.162516 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkxj\" (UniqueName: \"kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj\") pod \"maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.217661 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.217631 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:45.587763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:45.587733 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm"] Nov 02 22:21:45.589270 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:21:45.589242 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a13204_4924_41cb_9555_67f78ea9c1c7.slice/crio-c3cb0dd3d93029d76ba459fe1a58a60bac73b921aa81bb69975b63e265f6dcd3 WatchSource:0}: Error finding container c3cb0dd3d93029d76ba459fe1a58a60bac73b921aa81bb69975b63e265f6dcd3: Status 404 returned error can't find the container with id c3cb0dd3d93029d76ba459fe1a58a60bac73b921aa81bb69975b63e265f6dcd3 Nov 02 22:21:46.254782 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:46.254700 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6"] Nov 02 22:21:46.256921 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:46.256899 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-267d89-vx7b6"] Nov 02 22:21:46.488002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:46.487975 5091 generic.go:356] "Generic (PLEG): container finished" podID="f7a13204-4924-41cb-9555-67f78ea9c1c7" containerID="19acda02335e17187b6d5caad35f5eab97f0b827a928a0323d35f677536f9dde" exitCode=0 Nov 02 22:21:46.488156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:46.488006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" event={"ID":"f7a13204-4924-41cb-9555-67f78ea9c1c7","Type":"ContainerDied","Data":"19acda02335e17187b6d5caad35f5eab97f0b827a928a0323d35f677536f9dde"} Nov 02 22:21:46.488156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:46.488055 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" event={"ID":"f7a13204-4924-41cb-9555-67f78ea9c1c7","Type":"ContainerStarted","Data":"c3cb0dd3d93029d76ba459fe1a58a60bac73b921aa81bb69975b63e265f6dcd3"} Nov 02 22:21:46.975664 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:46.975633 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c" path="/var/lib/kubelet/pods/1ddc8a27-4e5f-4f17-987b-3dc8bc750c8c/volumes" Nov 02 22:21:47.045777 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.045747 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v"] Nov 02 22:21:47.047861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.047842 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574a4459-9v78v"] Nov 02 22:21:47.628454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.628427 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:47.730691 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.730646 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj\" (UniqueName: \"kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj\") pod \"f7a13204-4924-41cb-9555-67f78ea9c1c7\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " Nov 02 22:21:47.730874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.730788 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca\") pod \"f7a13204-4924-41cb-9555-67f78ea9c1c7\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " Nov 02 22:21:47.730874 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.730837 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca\") pod \"f7a13204-4924-41cb-9555-67f78ea9c1c7\" (UID: \"f7a13204-4924-41cb-9555-67f78ea9c1c7\") " Nov 02 22:21:47.731117 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.731095 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca" pod "f7a13204-4924-41cb-9555-67f78ea9c1c7" (UID: "f7a13204-4924-41cb-9555-67f78ea9c1c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:47.732867 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.732840 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca" pod "f7a13204-4924-41cb-9555-67f78ea9c1c7" (UID: "f7a13204-4924-41cb-9555-67f78ea9c1c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:47.732963 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.732921 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj" pod "f7a13204-4924-41cb-9555-67f78ea9c1c7" (UID: "f7a13204-4924-41cb-9555-67f78ea9c1c7"). InnerVolumeSpecName "kube-api-access-nbkxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:47.832410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.832345 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:47.832410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.832373 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a13204-4924-41cb-9555-67f78ea9c1c7-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:47.832410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:47.832383 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbkxj\" (UniqueName: \"kubernetes.io/projected/f7a13204-4924-41cb-9555-67f78ea9c1c7-kube-api-access-nbkxj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:48.045957 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.045920 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k"] Nov 02 22:21:48.048699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.048664 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d8006d2-jw26k"] Nov 02 22:21:48.500612 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.500583 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" Nov 02 22:21:48.500809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.500582 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm" event={"ID":"f7a13204-4924-41cb-9555-67f78ea9c1c7","Type":"ContainerDied","Data":"c3cb0dd3d93029d76ba459fe1a58a60bac73b921aa81bb69975b63e265f6dcd3"} Nov 02 22:21:48.500809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.500718 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3cb0dd3d93029d76ba459fe1a58a60bac73b921aa81bb69975b63e265f6dcd3" Nov 02 22:21:48.975838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.975803 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf21d29-2db2-4da0-9ac0-4277b385af3e" path="/var/lib/kubelet/pods/1bf21d29-2db2-4da0-9ac0-4277b385af3e/volumes" Nov 02 22:21:48.976224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:48.976207 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966f4d6d-e979-4964-9a64-3eaa0b446bc5" path="/var/lib/kubelet/pods/966f4d6d-e979-4964-9a64-3eaa0b446bc5/volumes" Nov 02 22:21:53.600621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.600582 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8"] Nov 02 22:21:53.601266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.601245 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f7a13204-4924-41cb-9555-67f78ea9c1c7" containerName="revoke-keys" Nov 02 22:21:53.601351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.601269 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a13204-4924-41cb-9555-67f78ea9c1c7" containerName="revoke-keys" Nov 02 22:21:53.601415 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.601405 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a13204-4924-41cb-9555-67f78ea9c1c7" containerName="revoke-keys" Nov 02 22:21:53.605875 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.605853 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.686239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.686210 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.686367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.686257 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.686367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.686356 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7db\" (UniqueName: \"kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.711224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.711182 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:53.711425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.711412 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:21:53.711425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.711417 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:21:53.711541 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.711433 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:21:53.713981 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.713961 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8"] Nov 02 22:21:53.787713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.787666 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.787879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.787733 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.787879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.787785 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7db\" (UniqueName: \"kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.788396 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.788373 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.790115 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.790095 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.843998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.843978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7db\" (UniqueName: \"kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db\") pod \"maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:53.919173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:53.919140 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:54.293851 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:54.293819 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8"] Nov 02 22:21:54.295426 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:21:54.295386 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928d5c61_cb98_40ea_af8d_6447a63a7c75.slice/crio-1c6c16baee6f7294ab3af8ebb8e1dc81327e4da4be200e289f37a7918b745c94 WatchSource:0}: Error finding container 1c6c16baee6f7294ab3af8ebb8e1dc81327e4da4be200e289f37a7918b745c94: Status 404 returned error can't find the container with id 1c6c16baee6f7294ab3af8ebb8e1dc81327e4da4be200e289f37a7918b745c94 Nov 02 22:21:54.537493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:54.537456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" event={"ID":"928d5c61-cb98-40ea-af8d-6447a63a7c75","Type":"ContainerStarted","Data":"1c6c16baee6f7294ab3af8ebb8e1dc81327e4da4be200e289f37a7918b745c94"} Nov 02 22:21:55.545433 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:55.545347 5091 generic.go:356] "Generic (PLEG): container finished" podID="928d5c61-cb98-40ea-af8d-6447a63a7c75" containerID="540469f2af56566cdfe9981ac641a5bb7cf49f5169c1051fc25438ffdf36c20e" exitCode=0 Nov 02 22:21:55.545851 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:55.545426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" event={"ID":"928d5c61-cb98-40ea-af8d-6447a63a7c75","Type":"ContainerDied","Data":"540469f2af56566cdfe9981ac641a5bb7cf49f5169c1051fc25438ffdf36c20e"} Nov 02 22:21:56.696256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.696231 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:56.714956 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.714930 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db\" (UniqueName: \"kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db\") pod \"928d5c61-cb98-40ea-af8d-6447a63a7c75\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " Nov 02 22:21:56.715091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.714964 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca\") pod \"928d5c61-cb98-40ea-af8d-6447a63a7c75\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " Nov 02 22:21:56.715091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.715050 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca\") pod \"928d5c61-cb98-40ea-af8d-6447a63a7c75\" (UID: \"928d5c61-cb98-40ea-af8d-6447a63a7c75\") " Nov 02 22:21:56.715404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.715381 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca" pod "928d5c61-cb98-40ea-af8d-6447a63a7c75" (UID: "928d5c61-cb98-40ea-af8d-6447a63a7c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:56.716993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.716971 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca" pod "928d5c61-cb98-40ea-af8d-6447a63a7c75" (UID: "928d5c61-cb98-40ea-af8d-6447a63a7c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:56.717075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.717021 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db" pod "928d5c61-cb98-40ea-af8d-6447a63a7c75" (UID: "928d5c61-cb98-40ea-af8d-6447a63a7c75"). InnerVolumeSpecName "kube-api-access-zc7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:56.816566 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.816490 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:56.816566 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.816526 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7db\" (UniqueName: \"kubernetes.io/projected/928d5c61-cb98-40ea-af8d-6447a63a7c75-kube-api-access-zc7db\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:56.816566 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:56.816542 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d5c61-cb98-40ea-af8d-6447a63a7c75-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:57.558766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:57.558736 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" Nov 02 22:21:57.558949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:57.558736 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8" event={"ID":"928d5c61-cb98-40ea-af8d-6447a63a7c75","Type":"ContainerDied","Data":"1c6c16baee6f7294ab3af8ebb8e1dc81327e4da4be200e289f37a7918b745c94"} Nov 02 22:21:57.558949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:21:57.558881 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c6c16baee6f7294ab3af8ebb8e1dc81327e4da4be200e289f37a7918b745c94" Nov 02 22:22:02.345508 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:02.345469 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm"] Nov 02 22:22:02.394979 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:02.394951 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-94a8a3-9v6dm"] Nov 02 22:22:02.975431 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:02.975397 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a13204-4924-41cb-9555-67f78ea9c1c7" path="/var/lib/kubelet/pods/f7a13204-4924-41cb-9555-67f78ea9c1c7/volumes" Nov 02 22:22:05.054137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:05.054105 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j"] Nov 02 22:22:05.158261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:05.158218 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq"] Nov 02 22:22:07.111461 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:07.111425 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:22:10.055322 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.055280 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" podUID="b295ea08-620f-4170-af89-fbc835dd9e15" containerName="payload-processing" containerID="cri-o://9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027" gracePeriod=25 Nov 02 22:22:10.123541 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.123514 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6"] Nov 02 22:22:10.123958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.123946 5091 cpu_manager.go:414] "containerMap: removing container" podUID="928d5c61-cb98-40ea-af8d-6447a63a7c75" containerName="revoke-keys" Nov 02 22:22:10.124012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.123960 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928d5c61-cb98-40ea-af8d-6447a63a7c75" containerName="revoke-keys" Nov 02 22:22:10.124054 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.124034 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="928d5c61-cb98-40ea-af8d-6447a63a7c75" containerName="revoke-keys" Nov 02 22:22:10.127191 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.127173 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.158803 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.158770 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" podUID="2dec3ed9-b13f-4524-a76a-07227b75457a" containerName="payload-pre-processing" containerID="cri-o://a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc" gracePeriod=25 Nov 02 22:22:10.181070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.181026 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:22:10.185287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.185260 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6"] Nov 02 22:22:10.230591 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.230557 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:22:10.230757 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.230651 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:22:10.230829 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.230788 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:22:10.246478 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.246450 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.246619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.246581 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gttb\" (UniqueName: \"kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.246713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.246646 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.310321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.310264 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 22:22:10.348089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.348056 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.348250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.348130 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gttb\" (UniqueName: \"kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.348250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.348191 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.348733 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.348712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.350533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.350507 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.404372 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.404347 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gttb\" (UniqueName: \"kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb\") pod \"maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.405078 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.405061 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 22:22:10.448884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.448860 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d\" (UniqueName: \"kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d\") pod \"b295ea08-620f-4170-af89-fbc835dd9e15\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " Nov 02 22:22:10.449031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.448922 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume\") pod \"2dec3ed9-b13f-4524-a76a-07227b75457a\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " Nov 02 22:22:10.449031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.448977 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk\" (UniqueName: \"kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk\") pod \"2dec3ed9-b13f-4524-a76a-07227b75457a\" (UID: \"2dec3ed9-b13f-4524-a76a-07227b75457a\") " Nov 02 22:22:10.449031 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.449019 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume\") pod \"b295ea08-620f-4170-af89-fbc835dd9e15\" (UID: \"b295ea08-620f-4170-af89-fbc835dd9e15\") " Nov 02 22:22:10.449421 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.449392 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume" pod "2dec3ed9-b13f-4524-a76a-07227b75457a" (UID: "2dec3ed9-b13f-4524-a76a-07227b75457a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:10.449475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.449451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume" pod "b295ea08-620f-4170-af89-fbc835dd9e15" (UID: "b295ea08-620f-4170-af89-fbc835dd9e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:10.451128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.451101 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk" pod "2dec3ed9-b13f-4524-a76a-07227b75457a" (UID: "2dec3ed9-b13f-4524-a76a-07227b75457a"). InnerVolumeSpecName "kube-api-access-56rhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:10.451128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.451109 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d" pod "b295ea08-620f-4170-af89-fbc835dd9e15" (UID: "b295ea08-620f-4170-af89-fbc835dd9e15"). InnerVolumeSpecName "kube-api-access-fmh2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:10.451128 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.451117 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:10.550236 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.550204 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec3ed9-b13f-4524-a76a-07227b75457a-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:10.550236 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.550235 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rhk\" (UniqueName: \"kubernetes.io/projected/2dec3ed9-b13f-4524-a76a-07227b75457a-kube-api-access-56rhk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:10.550429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.550249 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295ea08-620f-4170-af89-fbc835dd9e15-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:10.550429 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.550262 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh2d\" (UniqueName: \"kubernetes.io/projected/b295ea08-620f-4170-af89-fbc835dd9e15-kube-api-access-fmh2d\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:10.636570 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.636538 5091 generic.go:356] "Generic (PLEG): container finished" podID="2dec3ed9-b13f-4524-a76a-07227b75457a" containerID="a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc" exitCode=0 Nov 02 22:22:10.636781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.636586 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" Nov 02 22:22:10.636781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.636624 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" event={"ID":"2dec3ed9-b13f-4524-a76a-07227b75457a","Type":"ContainerDied","Data":"a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc"} Nov 02 22:22:10.636781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.636699 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq" event={"ID":"2dec3ed9-b13f-4524-a76a-07227b75457a","Type":"ContainerDied","Data":"9e77bd5a79c57df510bb9a35a00ca04c0b175696efd0a763e8fb33352f7d306e"} Nov 02 22:22:10.636781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.636733 5091 scope.go:122] "RemoveContainer" containerID="a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc" Nov 02 22:22:10.638075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.638054 5091 generic.go:356] "Generic (PLEG): container finished" podID="b295ea08-620f-4170-af89-fbc835dd9e15" containerID="9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027" exitCode=0 Nov 02 22:22:10.638166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.638083 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" event={"ID":"b295ea08-620f-4170-af89-fbc835dd9e15","Type":"ContainerDied","Data":"9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027"} Nov 02 22:22:10.638166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.638148 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" event={"ID":"b295ea08-620f-4170-af89-fbc835dd9e15","Type":"ContainerDied","Data":"de868b070aabf2da30d42c145664ebea8439afd6f6a1b1a9a7d95698755a298a"} Nov 02 22:22:10.638258 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.638100 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j" Nov 02 22:22:10.646146 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.646129 5091 scope.go:122] "RemoveContainer" containerID="a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc" Nov 02 22:22:10.646417 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:10.646394 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc\": container with ID starting with a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc not found: ID does not exist" containerID="a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc" Nov 02 22:22:10.646494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.646418 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc"} err="failed to get container status \"a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc\": rpc error: code = NotFound desc = could not find container \"a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc\": container with ID starting with a2dd2883e295331cf8ca4a9444e12ab6d3957108866b0175b3808e44eb2dadfc not found: ID does not exist" Nov 02 22:22:10.646494 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.646434 5091 scope.go:122] "RemoveContainer" containerID="9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027" Nov 02 22:22:10.655109 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.655092 5091 scope.go:122] "RemoveContainer" containerID="9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027" Nov 02 22:22:10.655355 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:10.655332 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027\": container with ID starting with 9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027 not found: ID does not exist" containerID="9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027" Nov 02 22:22:10.655423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.655361 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027"} err="failed to get container status \"9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027\": rpc error: code = NotFound desc = could not find container \"9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027\": container with ID starting with 9a18cd7129cd962444e2114c4a41f2b2b97271cda4142526a46cf98ebd53e027 not found: ID does not exist" Nov 02 22:22:10.798456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.798425 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq"] Nov 02 22:22:10.800710 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.800685 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10fdd5-5d45d4df79-pcrfq"] Nov 02 22:22:10.819077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.819057 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6"] Nov 02 22:22:10.821436 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:22:10.821417 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6a4bca_b04d_4458_a9d4_8e8e741b5ca2.slice/crio-9f7f61d2e9f28bf694251bd6d5a4e8f8c68d1c5ac58679c2f8d2e0ced02dfcd9 WatchSource:0}: Error finding container 9f7f61d2e9f28bf694251bd6d5a4e8f8c68d1c5ac58679c2f8d2e0ced02dfcd9: Status 404 returned error can't find the container with id 9f7f61d2e9f28bf694251bd6d5a4e8f8c68d1c5ac58679c2f8d2e0ced02dfcd9 Nov 02 22:22:10.908905 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.908844 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j"] Nov 02 22:22:10.911511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.911490 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10fdd5-5694ddb679-8qz8j"] Nov 02 22:22:10.979000 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.976782 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dec3ed9-b13f-4524-a76a-07227b75457a" path="/var/lib/kubelet/pods/2dec3ed9-b13f-4524-a76a-07227b75457a/volumes" Nov 02 22:22:10.979000 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:10.977529 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b295ea08-620f-4170-af89-fbc835dd9e15" path="/var/lib/kubelet/pods/b295ea08-620f-4170-af89-fbc835dd9e15/volumes" Nov 02 22:22:11.594262 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594236 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j"] Nov 02 22:22:11.594695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594664 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2dec3ed9-b13f-4524-a76a-07227b75457a" containerName="payload-pre-processing" Nov 02 22:22:11.594749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594694 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dec3ed9-b13f-4524-a76a-07227b75457a" containerName="payload-pre-processing" Nov 02 22:22:11.594749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594721 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b295ea08-620f-4170-af89-fbc835dd9e15" containerName="payload-processing" Nov 02 22:22:11.594749 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594726 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b295ea08-620f-4170-af89-fbc835dd9e15" containerName="payload-processing" Nov 02 22:22:11.594866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594835 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dec3ed9-b13f-4524-a76a-07227b75457a" containerName="payload-pre-processing" Nov 02 22:22:11.594866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.594847 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b295ea08-620f-4170-af89-fbc835dd9e15" containerName="payload-processing" Nov 02 22:22:11.598320 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.598297 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.645683 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.645643 5091 generic.go:356] "Generic (PLEG): container finished" podID="0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" containerID="d94b14830cd3467bb03fce856ffcecfb2bf0b24376fca6b9fc27305498d23ca4" exitCode=0 Nov 02 22:22:11.645816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.645729 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" event={"ID":"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2","Type":"ContainerDied","Data":"d94b14830cd3467bb03fce856ffcecfb2bf0b24376fca6b9fc27305498d23ca4"} Nov 02 22:22:11.645816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.645772 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" event={"ID":"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2","Type":"ContainerStarted","Data":"9f7f61d2e9f28bf694251bd6d5a4e8f8c68d1c5ac58679c2f8d2e0ced02dfcd9"} Nov 02 22:22:11.654941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.654918 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j"] Nov 02 22:22:11.660734 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.660707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.660886 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.660825 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6zd\" (UniqueName: \"kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.660886 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.660869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.762081 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.762054 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6zd\" (UniqueName: \"kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.762235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.762103 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.762235 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.762167 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.762840 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.762817 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.764773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.764747 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.819994 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.819938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6zd\" (UniqueName: \"kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd\") pod \"maas-api-revoke-sub-037110d3-d4k6j\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:11.937328 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:11.937300 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:12.307356 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.307329 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j"] Nov 02 22:22:12.309291 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:22:12.309258 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-ba6a727d3e192062fbfc9ae80b5c85f216ae1c92b33e2e89d2a5933c30e17a3f WatchSource:0}: Error finding container ba6a727d3e192062fbfc9ae80b5c85f216ae1c92b33e2e89d2a5933c30e17a3f: Status 404 returned error can't find the container with id ba6a727d3e192062fbfc9ae80b5c85f216ae1c92b33e2e89d2a5933c30e17a3f Nov 02 22:22:12.654861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.654824 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" event={"ID":"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1","Type":"ContainerStarted","Data":"19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154"} Nov 02 22:22:12.655289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.654879 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" event={"ID":"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1","Type":"ContainerStarted","Data":"ba6a727d3e192062fbfc9ae80b5c85f216ae1c92b33e2e89d2a5933c30e17a3f"} Nov 02 22:22:12.767996 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.767931 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" podStartSLOduration=1.7679133839999999 podStartE2EDuration="1.767913384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:12.767566718 +0000 UTC m=+2879.929661837" watchObservedRunningTime="2026-11-02 22:22:12.767913384 +0000 UTC m=+2879.930008505" Nov 02 22:22:12.797395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.797371 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:12.872382 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.872359 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb\" (UniqueName: \"kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb\") pod \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " Nov 02 22:22:12.872506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.872481 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca\") pod \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " Nov 02 22:22:12.872554 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.872508 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca\") pod \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\" (UID: \"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2\") " Nov 02 22:22:12.872844 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.872820 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca" pod "0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" (UID: "0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:12.874484 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.874461 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb" pod "0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" (UID: "0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2"). InnerVolumeSpecName "kube-api-access-8gttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:12.874568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.874500 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca" pod "0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" (UID: "0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:12.973665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.973633 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gttb\" (UniqueName: \"kubernetes.io/projected/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-kube-api-access-8gttb\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:12.973665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.973665 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:12.973884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:12.973695 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.663216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:13.663181 5091 generic.go:356] "Generic (PLEG): container finished" podID="e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" containerID="19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154" exitCode=0 Nov 02 22:22:13.663698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:13.663267 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" event={"ID":"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1","Type":"ContainerDied","Data":"19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154"} Nov 02 22:22:13.664661 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:13.664647 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" Nov 02 22:22:13.664753 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:13.664710 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6" event={"ID":"0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2","Type":"ContainerDied","Data":"9f7f61d2e9f28bf694251bd6d5a4e8f8c68d1c5ac58679c2f8d2e0ced02dfcd9"} Nov 02 22:22:13.664797 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:13.664756 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7f61d2e9f28bf694251bd6d5a4e8f8c68d1c5ac58679c2f8d2e0ced02dfcd9" Nov 02 22:22:14.604814 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.604781 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h"] Nov 02 22:22:14.654440 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.654410 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm"] Nov 02 22:22:14.656605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.656568 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8"] Nov 02 22:22:14.805074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.805052 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:14.891371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.891343 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca\") pod \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " Nov 02 22:22:14.891533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.891378 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd\" (UniqueName: \"kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd\") pod \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " Nov 02 22:22:14.891533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.891453 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca\") pod \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\" (UID: \"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1\") " Nov 02 22:22:14.891761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.891740 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca" pod "e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" (UID: "e04e2c12-0fa1-4b85-9d21-6b6c98388ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:14.893511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.893483 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca" pod "e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" (UID: "e04e2c12-0fa1-4b85-9d21-6b6c98388ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:14.893595 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.893548 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd" pod "e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" (UID: "e04e2c12-0fa1-4b85-9d21-6b6c98388ff1"). InnerVolumeSpecName "kube-api-access-ws6zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:14.992475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.992446 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6zd\" (UniqueName: \"kubernetes.io/projected/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-kube-api-access-ws6zd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.992475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.992471 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.992624 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:14.992481 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:15.678686 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:15.678639 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" Nov 02 22:22:15.678862 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:15.678691 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j" event={"ID":"e04e2c12-0fa1-4b85-9d21-6b6c98388ff1","Type":"ContainerDied","Data":"ba6a727d3e192062fbfc9ae80b5c85f216ae1c92b33e2e89d2a5933c30e17a3f"} Nov 02 22:22:15.678862 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:15.678770 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6a727d3e192062fbfc9ae80b5c85f216ae1c92b33e2e89d2a5933c30e17a3f" Nov 02 22:22:16.465418 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:16.465379 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:16.468371 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:16.468340 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:16.468551 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:16.468523 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:18.949108 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:18.949062 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:21.106160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:21.106128 5091 scope.go:122] "RemoveContainer" containerID="427e510c1d485299f33fd48da66381a876775330f924da19bd409e4b21903ee3" Nov 02 22:22:21.582149 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:21.582113 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc"] Nov 02 22:22:21.688091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:21.688052 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7"] Nov 02 22:22:26.246665 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.246628 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8"] Nov 02 22:22:26.247083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.247033 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" containerName="revoke-keys" Nov 02 22:22:26.247083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.247046 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" containerName="revoke-keys" Nov 02 22:22:26.247083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.247077 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" containerName="revoke-keys" Nov 02 22:22:26.247083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.247082 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" containerName="revoke-keys" Nov 02 22:22:26.247251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.247147 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" containerName="revoke-keys" Nov 02 22:22:26.247251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.247161 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" containerName="revoke-keys" Nov 02 22:22:26.251835 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.251819 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.301869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.301834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bl78\" (UniqueName: \"kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.301997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.301881 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.301997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.301993 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.307321 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.307133 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8"] Nov 02 22:22:26.353423 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.353392 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:22:26.353564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.353488 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:22:26.353564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.353506 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:22:26.353564 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.353556 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:22:26.403027 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.402991 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bl78\" (UniqueName: \"kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.403170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.403032 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.403170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.403082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.403740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.403711 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.405452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.405431 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.459018 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.458987 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bl78\" (UniqueName: \"kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78\") pod \"maas-api-revoke-sub-786b24e5-ts2r8\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.524016 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:26.523938 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:26.564397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.564373 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:26.583014 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.582964 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" podUID="c35d741f-a717-424d-a22e-5fd1b834e96f" containerName="payload-processing" containerID="cri-o://09e33b3dd034c9b853d4937bc2f7825219af86a124871884b9d8749f70313728" gracePeriod=25 Nov 02 22:22:26.688587 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.688537 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerName="payload-pre-processing" containerID="cri-o://0e68cd9fb7faa7fac5fbe1740bc9834a236319ebad6bfecb0b8d8edc2fe5e23c" gracePeriod=25 Nov 02 22:22:26.750760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.750724 5091 generic.go:356] "Generic (PLEG): container finished" podID="c35d741f-a717-424d-a22e-5fd1b834e96f" containerID="09e33b3dd034c9b853d4937bc2f7825219af86a124871884b9d8749f70313728" exitCode=0 Nov 02 22:22:26.750760 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.750799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" event={"ID":"c35d741f-a717-424d-a22e-5fd1b834e96f","Type":"ContainerDied","Data":"09e33b3dd034c9b853d4937bc2f7825219af86a124871884b9d8749f70313728"} Nov 02 22:22:26.825661 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.825635 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 22:22:26.908169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.908138 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk\" (UniqueName: \"kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk\") pod \"c35d741f-a717-424d-a22e-5fd1b834e96f\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " Nov 02 22:22:26.908338 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.908184 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume\") pod \"c35d741f-a717-424d-a22e-5fd1b834e96f\" (UID: \"c35d741f-a717-424d-a22e-5fd1b834e96f\") " Nov 02 22:22:26.908594 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.908562 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume" pod "c35d741f-a717-424d-a22e-5fd1b834e96f" (UID: "c35d741f-a717-424d-a22e-5fd1b834e96f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:26.910283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.910264 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk" pod "c35d741f-a717-424d-a22e-5fd1b834e96f" (UID: "c35d741f-a717-424d-a22e-5fd1b834e96f"). InnerVolumeSpecName "kube-api-access-67wsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:26.935532 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:26.935509 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8"] Nov 02 22:22:26.936815 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:22:26.936791 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b65b31_a557_4e4e_9dc8_589ddc9a9002.slice/crio-5d3a6f86f20c2367e319c131c36e2fde7ce3168d9a6d9389c17f2d005f61cfc5 WatchSource:0}: Error finding container 5d3a6f86f20c2367e319c131c36e2fde7ce3168d9a6d9389c17f2d005f61cfc5: Status 404 returned error can't find the container with id 5d3a6f86f20c2367e319c131c36e2fde7ce3168d9a6d9389c17f2d005f61cfc5 Nov 02 22:22:27.009159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.009127 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wsk\" (UniqueName: \"kubernetes.io/projected/c35d741f-a717-424d-a22e-5fd1b834e96f-kube-api-access-67wsk\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:27.009159 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.009153 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35d741f-a717-424d-a22e-5fd1b834e96f-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:27.757374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.757348 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" Nov 02 22:22:27.757374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.757356 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc" event={"ID":"c35d741f-a717-424d-a22e-5fd1b834e96f","Type":"ContainerDied","Data":"ba283de78fe26baea0ec0a2251c1cf661332c4202594cc37fe9cedd858c3560f"} Nov 02 22:22:27.757903 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.757417 5091 scope.go:122] "RemoveContainer" containerID="09e33b3dd034c9b853d4937bc2f7825219af86a124871884b9d8749f70313728" Nov 02 22:22:27.759066 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.759044 5091 generic.go:356] "Generic (PLEG): container finished" podID="07b65b31-a557-4e4e-9dc8-589ddc9a9002" containerID="eb6235ae91f3b312b72f4d5433f2e383c27038bc69096d7ef677ec431fa94302" exitCode=0 Nov 02 22:22:27.759171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.759127 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" event={"ID":"07b65b31-a557-4e4e-9dc8-589ddc9a9002","Type":"ContainerDied","Data":"eb6235ae91f3b312b72f4d5433f2e383c27038bc69096d7ef677ec431fa94302"} Nov 02 22:22:27.759220 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:27.759177 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" event={"ID":"07b65b31-a557-4e4e-9dc8-589ddc9a9002","Type":"ContainerStarted","Data":"5d3a6f86f20c2367e319c131c36e2fde7ce3168d9a6d9389c17f2d005f61cfc5"} Nov 02 22:22:28.034215 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:28.034131 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc"] Nov 02 22:22:28.038051 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:28.038024 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9cc56b-8f67bf679-n72vc"] Nov 02 22:22:28.904619 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:28.904593 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:28.975416 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:28.975388 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35d741f-a717-424d-a22e-5fd1b834e96f" path="/var/lib/kubelet/pods/c35d741f-a717-424d-a22e-5fd1b834e96f/volumes" Nov 02 22:22:29.025309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.025236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca\") pod \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " Nov 02 22:22:29.025309 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.025294 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca\") pod \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " Nov 02 22:22:29.025556 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.025364 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78\" (UniqueName: \"kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78\") pod \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\" (UID: \"07b65b31-a557-4e4e-9dc8-589ddc9a9002\") " Nov 02 22:22:29.025620 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.025563 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca" pod "07b65b31-a557-4e4e-9dc8-589ddc9a9002" (UID: "07b65b31-a557-4e4e-9dc8-589ddc9a9002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:29.025781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.025738 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:29.028250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.028223 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78" pod "07b65b31-a557-4e4e-9dc8-589ddc9a9002" (UID: "07b65b31-a557-4e4e-9dc8-589ddc9a9002"). InnerVolumeSpecName "kube-api-access-4bl78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:29.028380 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.028285 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca" pod "07b65b31-a557-4e4e-9dc8-589ddc9a9002" (UID: "07b65b31-a557-4e4e-9dc8-589ddc9a9002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:29.126941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.126907 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b65b31-a557-4e4e-9dc8-589ddc9a9002-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:29.126941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.126940 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bl78\" (UniqueName: \"kubernetes.io/projected/07b65b31-a557-4e4e-9dc8-589ddc9a9002-kube-api-access-4bl78\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:29.625231 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.625169 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" containerID="cri-o://be54c3746766efacdcfdd160499fef54f2c3213f36e66114e22bbbe0fe1ea987" gracePeriod=45 Nov 02 22:22:29.675526 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.675491 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" containerID="cri-o://c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca" gracePeriod=45 Nov 02 22:22:29.675661 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.675636 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" containerID="cri-o://dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269" gracePeriod=45 Nov 02 22:22:29.688541 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.688513 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": dial tcp 10.132.0.109:8000: connect: connection refused" Nov 02 22:22:29.773289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.773265 5091 generic.go:356] "Generic (PLEG): container finished" podID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerID="be54c3746766efacdcfdd160499fef54f2c3213f36e66114e22bbbe0fe1ea987" exitCode=0 Nov 02 22:22:29.773402 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.773337 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" event={"ID":"59f8c12f-cbce-4387-a9e6-f3a5e22796ee","Type":"ContainerDied","Data":"be54c3746766efacdcfdd160499fef54f2c3213f36e66114e22bbbe0fe1ea987"} Nov 02 22:22:29.774590 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.774574 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" Nov 02 22:22:29.774722 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.774595 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8" event={"ID":"07b65b31-a557-4e4e-9dc8-589ddc9a9002","Type":"ContainerDied","Data":"5d3a6f86f20c2367e319c131c36e2fde7ce3168d9a6d9389c17f2d005f61cfc5"} Nov 02 22:22:29.774722 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.774629 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d3a6f86f20c2367e319c131c36e2fde7ce3168d9a6d9389c17f2d005f61cfc5" Nov 02 22:22:29.939323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.939297 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 22:22:29.970041 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.970014 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 22:22:29.972247 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:29.972232 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 22:22:30.032430 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032395 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8\" (UniqueName: \"kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8\") pod \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " Nov 02 22:22:30.032610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032437 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs\") pod \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " Nov 02 22:22:30.032610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache\") pod \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " Nov 02 22:22:30.032610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032503 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home\") pod \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " Nov 02 22:22:30.032610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2\" (UniqueName: \"kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2\") pod \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " Nov 02 22:22:30.032610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir\") pod \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " Nov 02 22:22:30.032610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032602 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache\") pod \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032637 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache\") pod \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032719 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87\" (UniqueName: \"kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87\") pod \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032761 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm\") pod \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032785 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home\") pod \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032807 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache" pod "59f8c12f-cbce-4387-a9e6-f3a5e22796ee" (UID: "59f8c12f-cbce-4387-a9e6-f3a5e22796ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032826 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home" pod "dc9c8fbb-ef28-4236-a453-343ce87b22bc" (UID: "dc9c8fbb-ef28-4236-a453-343ce87b22bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032864 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs\") pod \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm\") pod \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032925 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs\") pod \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032950 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm\") pod \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " Nov 02 22:22:30.032992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.032984 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir\") pod \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\" (UID: \"dc9c8fbb-ef28-4236-a453-343ce87b22bc\") " Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033024 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir\") pod \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\" (UID: \"59f8c12f-cbce-4387-a9e6-f3a5e22796ee\") " Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033053 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home\") pod \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\" (UID: \"b425e9fb-038d-4b47-859c-d5cf7c2c22dc\") " Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033048 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir" pod "b425e9fb-038d-4b47-859c-d5cf7c2c22dc" (UID: "b425e9fb-038d-4b47-859c-d5cf7c2c22dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033108 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache" pod "b425e9fb-038d-4b47-859c-d5cf7c2c22dc" (UID: "b425e9fb-038d-4b47-859c-d5cf7c2c22dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033403 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache" pod "dc9c8fbb-ef28-4236-a453-343ce87b22bc" (UID: "dc9c8fbb-ef28-4236-a453-343ce87b22bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033410 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033446 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033464 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.033621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033480 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.034171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033630 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home" pod "b425e9fb-038d-4b47-859c-d5cf7c2c22dc" (UID: "b425e9fb-038d-4b47-859c-d5cf7c2c22dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.034171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033749 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir" pod "dc9c8fbb-ef28-4236-a453-343ce87b22bc" (UID: "dc9c8fbb-ef28-4236-a453-343ce87b22bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.034171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.033857 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir" pod "59f8c12f-cbce-4387-a9e6-f3a5e22796ee" (UID: "59f8c12f-cbce-4387-a9e6-f3a5e22796ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.035537 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.035488 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm" pod "59f8c12f-cbce-4387-a9e6-f3a5e22796ee" (UID: "59f8c12f-cbce-4387-a9e6-f3a5e22796ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.035802 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.035762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2" pod "59f8c12f-cbce-4387-a9e6-f3a5e22796ee" (UID: "59f8c12f-cbce-4387-a9e6-f3a5e22796ee"). InnerVolumeSpecName "kube-api-access-89qw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:30.035894 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.035798 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home" pod "59f8c12f-cbce-4387-a9e6-f3a5e22796ee" (UID: "59f8c12f-cbce-4387-a9e6-f3a5e22796ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.037156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.037135 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87" pod "dc9c8fbb-ef28-4236-a453-343ce87b22bc" (UID: "dc9c8fbb-ef28-4236-a453-343ce87b22bc"). InnerVolumeSpecName "kube-api-access-sjz87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:30.037614 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.037590 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs" pod "dc9c8fbb-ef28-4236-a453-343ce87b22bc" (UID: "dc9c8fbb-ef28-4236-a453-343ce87b22bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:30.037724 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.037632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs" pod "59f8c12f-cbce-4387-a9e6-f3a5e22796ee" (UID: "59f8c12f-cbce-4387-a9e6-f3a5e22796ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:30.037961 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.037946 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm" pod "dc9c8fbb-ef28-4236-a453-343ce87b22bc" (UID: "dc9c8fbb-ef28-4236-a453-343ce87b22bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.038162 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.038141 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs" pod "b425e9fb-038d-4b47-859c-d5cf7c2c22dc" (UID: "b425e9fb-038d-4b47-859c-d5cf7c2c22dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:30.038263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.038247 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8" pod "b425e9fb-038d-4b47-859c-d5cf7c2c22dc" (UID: "b425e9fb-038d-4b47-859c-d5cf7c2c22dc"). InnerVolumeSpecName "kube-api-access-ggpd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:30.038348 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.038334 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm" pod "b425e9fb-038d-4b47-859c-d5cf7c2c22dc" (UID: "b425e9fb-038d-4b47-859c-d5cf7c2c22dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:30.134696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134652 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qw2\" (UniqueName: \"kubernetes.io/projected/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-kube-api-access-89qw2\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134695 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134705 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjz87\" (UniqueName: \"kubernetes.io/projected/dc9c8fbb-ef28-4236-a453-343ce87b22bc-kube-api-access-sjz87\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134714 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134724 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134732 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134742 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134750 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134757 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134764 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134771 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f8c12f-cbce-4387-a9e6-f3a5e22796ee-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134779 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134787 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpd8\" (UniqueName: \"kubernetes.io/projected/b425e9fb-038d-4b47-859c-d5cf7c2c22dc-kube-api-access-ggpd8\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.134943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.134795 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9c8fbb-ef28-4236-a453-343ce87b22bc-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.781397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.781364 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" Nov 02 22:22:30.781397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.781376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h" event={"ID":"59f8c12f-cbce-4387-a9e6-f3a5e22796ee","Type":"ContainerDied","Data":"e9414813b3b47d13d7f462406bb75eb1608f18f78c48ef772bbeac5dce24df59"} Nov 02 22:22:30.781656 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.781420 5091 scope.go:122] "RemoveContainer" containerID="be54c3746766efacdcfdd160499fef54f2c3213f36e66114e22bbbe0fe1ea987" Nov 02 22:22:30.782863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.782842 5091 generic.go:356] "Generic (PLEG): container finished" podID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerID="c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca" exitCode=0 Nov 02 22:22:30.783077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.782885 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" Nov 02 22:22:30.783077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.782924 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" event={"ID":"dc9c8fbb-ef28-4236-a453-343ce87b22bc","Type":"ContainerDied","Data":"c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca"} Nov 02 22:22:30.783077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.782961 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8" event={"ID":"dc9c8fbb-ef28-4236-a453-343ce87b22bc","Type":"ContainerDied","Data":"50177b816a2aeabaa33c3cdbe13edeefd4ce3849a5b48da9c2e501e7c854fb01"} Nov 02 22:22:30.784399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.784376 5091 generic.go:356] "Generic (PLEG): container finished" podID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerID="dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269" exitCode=0 Nov 02 22:22:30.784522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.784424 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" Nov 02 22:22:30.784522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.784427 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" event={"ID":"b425e9fb-038d-4b47-859c-d5cf7c2c22dc","Type":"ContainerDied","Data":"dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269"} Nov 02 22:22:30.784611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.784534 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm" event={"ID":"b425e9fb-038d-4b47-859c-d5cf7c2c22dc","Type":"ContainerDied","Data":"75ff6e3164d515bd1409b21035b2e5fcc2f6af52f386bc6d042ea169de1f381e"} Nov 02 22:22:30.791740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.791664 5091 scope.go:122] "RemoveContainer" containerID="c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca" Nov 02 22:22:30.801693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.801624 5091 scope.go:122] "RemoveContainer" containerID="c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca" Nov 02 22:22:30.801936 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:30.801915 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca\": container with ID starting with c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca not found: ID does not exist" containerID="c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca" Nov 02 22:22:30.802038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.801941 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca"} err="failed to get container status \"c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca\": rpc error: code = NotFound desc = could not find container \"c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca\": container with ID starting with c9a0fcdb7a5f3d7d7af58e7187abced6147f52d25414f30575cf475a37e3dbca not found: ID does not exist" Nov 02 22:22:30.802038 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.801959 5091 scope.go:122] "RemoveContainer" containerID="dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269" Nov 02 22:22:30.810105 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.810080 5091 scope.go:122] "RemoveContainer" containerID="dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269" Nov 02 22:22:30.810346 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:30.810329 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269\": container with ID starting with dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269 not found: ID does not exist" containerID="dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269" Nov 02 22:22:30.810399 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.810353 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269"} err="failed to get container status \"dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269\": rpc error: code = NotFound desc = could not find container \"dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269\": container with ID starting with dfbfefa291ee0aec47454356e27273f71f6ba0dbb6810ecfcf139a165e249269 not found: ID does not exist" Nov 02 22:22:30.943715 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.943682 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h"] Nov 02 22:22:30.993099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:30.993072 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-2-w2-9cc56b-kserve-6d8784c56-ltv2h"] Nov 02 22:22:31.057730 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:31.057602 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8"] Nov 02 22:22:31.060436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:31.060406 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/simulator-w2-9cc56b-kserve-68b98f446f-cf4j8"] Nov 02 22:22:31.169885 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:31.169849 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm"] Nov 02 22:22:31.173288 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:31.173258 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/distinct-w2-9cc56b-kserve-6ccdd9ff87-txlbm"] Nov 02 22:22:31.488666 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:31.488627 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:22:32.976496 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:32.976466 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" path="/var/lib/kubelet/pods/59f8c12f-cbce-4387-a9e6-f3a5e22796ee/volumes" Nov 02 22:22:32.976916 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:32.976901 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" path="/var/lib/kubelet/pods/b425e9fb-038d-4b47-859c-d5cf7c2c22dc/volumes" Nov 02 22:22:32.977276 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:32.977263 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" path="/var/lib/kubelet/pods/dc9c8fbb-ef28-4236-a453-343ce87b22bc/volumes" Nov 02 22:22:33.160627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:33.160589 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8"] Nov 02 22:22:33.209909 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:33.209872 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-10fdd5-lrkd8"] Nov 02 22:22:33.983968 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:33.983935 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:34.976055 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:34.976022 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928d5c61-cb98-40ea-af8d-6447a63a7c75" path="/var/lib/kubelet/pods/928d5c61-cb98-40ea-af8d-6447a63a7c75/volumes" Nov 02 22:22:36.539937 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:36.539899 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:37.113150 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:37.113112 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:22:41.488662 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:41.488627 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:22:42.187143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:42.187102 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6"] Nov 02 22:22:42.236578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:42.236544 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9cc56b-86cl6"] Nov 02 22:22:42.976498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:42.976458 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2" path="/var/lib/kubelet/pods/0e6a4bca-b04d-4458-a9d4-8e8e741b5ca2/volumes" Nov 02 22:22:46.593109 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:46.593078 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:46.885374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:46.885344 5091 generic.go:356] "Generic (PLEG): container finished" podID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerID="0e68cd9fb7faa7fac5fbe1740bc9834a236319ebad6bfecb0b8d8edc2fe5e23c" exitCode=0 Nov 02 22:22:46.885546 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:46.885397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" event={"ID":"5f0ccbd5-e95f-43bb-8811-961a812f2bc7","Type":"ContainerDied","Data":"0e68cd9fb7faa7fac5fbe1740bc9834a236319ebad6bfecb0b8d8edc2fe5e23c"} Nov 02 22:22:46.949245 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:46.949217 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 22:22:47.099692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.099631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj\" (UniqueName: \"kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj\") pod \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " Nov 02 22:22:47.099891 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.099749 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume\") pod \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\" (UID: \"5f0ccbd5-e95f-43bb-8811-961a812f2bc7\") " Nov 02 22:22:47.100374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.100155 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume" pod "5f0ccbd5-e95f-43bb-8811-961a812f2bc7" (UID: "5f0ccbd5-e95f-43bb-8811-961a812f2bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:47.100374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.100262 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-plugins-config-volume\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:47.101958 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.101937 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj" pod "5f0ccbd5-e95f-43bb-8811-961a812f2bc7" (UID: "5f0ccbd5-e95f-43bb-8811-961a812f2bc7"). InnerVolumeSpecName "kube-api-access-6zxzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:47.201278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.201240 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxzj\" (UniqueName: \"kubernetes.io/projected/5f0ccbd5-e95f-43bb-8811-961a812f2bc7-kube-api-access-6zxzj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:47.892908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.892876 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" Nov 02 22:22:47.893359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.892911 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7" event={"ID":"5f0ccbd5-e95f-43bb-8811-961a812f2bc7","Type":"ContainerDied","Data":"c32531b127f069c8e541346fe44ff68d163c5f49f5c02d2d801d60b21bffe708"} Nov 02 22:22:47.893359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:47.892970 5091 scope.go:122] "RemoveContainer" containerID="0e68cd9fb7faa7fac5fbe1740bc9834a236319ebad6bfecb0b8d8edc2fe5e23c" Nov 02 22:22:48.055242 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:48.055211 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7"] Nov 02 22:22:48.105208 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:48.105179 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9cc56b-884555655-9fhl7"] Nov 02 22:22:48.955172 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:48.955138 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:48.978522 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:48.978489 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" path="/var/lib/kubelet/pods/5f0ccbd5-e95f-43bb-8811-961a812f2bc7/volumes" Nov 02 22:22:49.001689 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:49.001645 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8"] Nov 02 22:22:49.001888 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:49.001858 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" containerID="cri-o://814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d" gracePeriod=30 Nov 02 22:22:50.028947 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:50.028911 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:22:50.029325 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:50.028981 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:22:54.261744 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.261716 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 22:22:54.366985 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.366908 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.366985 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.366965 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.366993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367010 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367030 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367051 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367141 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367180 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo\") pod \"de1dd20c-2f12-4faf-9425-b2cf70938868\" (UID: \"de1dd20c-2f12-4faf-9425-b2cf70938868\") " Nov 02 22:22:54.367621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367347 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:54.367621 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367557 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:54.367742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367622 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1dd20c-2f12-4faf-9425-b2cf70938868-istiod-ca-cert\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.367742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367349 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:54.367742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367698 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:54.367851 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.367746 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:54.369919 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.369891 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:22:54.370034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.369925 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:54.370034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.369940 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "kube-api-access-vzqn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:54.370034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.369983 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token" pod "de1dd20c-2f12-4faf-9425-b2cf70938868" (UID: "de1dd20c-2f12-4faf-9425-b2cf70938868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:54.468226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468191 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-envoy\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468220 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468231 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-data\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468242 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqn5\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-kube-api-access-vzqn5\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468254 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-token\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468262 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-credential-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468271 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1dd20c-2f12-4faf-9425-b2cf70938868-workload-socket\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.468450 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.468280 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1dd20c-2f12-4faf-9425-b2cf70938868-istio-podinfo\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.934332 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.934301 5091 generic.go:356] "Generic (PLEG): container finished" podID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerID="814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d" exitCode=0 Nov 02 22:22:54.934332 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.934337 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" event={"ID":"de1dd20c-2f12-4faf-9425-b2cf70938868","Type":"ContainerDied","Data":"814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d"} Nov 02 22:22:54.934554 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.934351 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" Nov 02 22:22:54.934554 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.934403 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8" event={"ID":"de1dd20c-2f12-4faf-9425-b2cf70938868","Type":"ContainerDied","Data":"5908d55db302f73670dc05ede64ae1b557e15e60269dd07a89c00c2200473446"} Nov 02 22:22:54.934554 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.934425 5091 scope.go:122] "RemoveContainer" containerID="814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d" Nov 02 22:22:54.944578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.944560 5091 scope.go:122] "RemoveContainer" containerID="814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d" Nov 02 22:22:54.944851 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:54.944832 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d\": container with ID starting with 814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d not found: ID does not exist" containerID="814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d" Nov 02 22:22:54.944931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:54.944860 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d"} err="failed to get container status \"814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d\": rpc error: code = NotFound desc = could not find container \"814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d\": container with ID starting with 814aa163bef50f0eab293366673b82380330326da71a5c92c71a990d518b158d not found: ID does not exist" Nov 02 22:22:55.097662 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:55.097626 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8"] Nov 02 22:22:55.146981 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:55.146950 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-62hv8"] Nov 02 22:22:56.605304 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:22:56.605267 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:22:56.974922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:22:56.974887 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" path="/var/lib/kubelet/pods/de1dd20c-2f12-4faf-9425-b2cf70938868/volumes" Nov 02 22:23:03.986257 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:23:03.986225 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:23:06.617280 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:23:06.617241 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2c12_0fa1_4b85_9d21_6b6c98388ff1.slice/crio-19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:23:07.111140 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:23:07.111106 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:23:21.151992 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:21.151956 5091 scope.go:122] "RemoveContainer" containerID="2ca8c2673817f55bd5f6b3bdc7e3320cbb96be7bae979dc6c38995857c075112" Nov 02 22:23:21.161210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:21.161188 5091 scope.go:122] "RemoveContainer" containerID="59e10a82ec47c1ce79c7b8cb1196856197f2aa1c6f820e5eeb4de3df270daee8" Nov 02 22:23:34.878742 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.878700 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss"] Nov 02 22:23:34.879275 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879254 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" Nov 02 22:23:34.879359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879279 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" Nov 02 22:23:34.879359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879298 5091 cpu_manager.go:414] "containerMap: removing container" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" Nov 02 22:23:34.879359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879306 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" Nov 02 22:23:34.879359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879323 5091 cpu_manager.go:414] "containerMap: removing container" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" Nov 02 22:23:34.879359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879331 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" Nov 02 22:23:34.879359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879353 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerName="payload-pre-processing" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879361 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerName="payload-pre-processing" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879377 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c35d741f-a717-424d-a22e-5fd1b834e96f" containerName="payload-processing" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879396 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35d741f-a717-424d-a22e-5fd1b834e96f" containerName="payload-processing" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879419 5091 cpu_manager.go:414] "containerMap: removing container" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879426 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879445 5091 cpu_manager.go:414] "containerMap: removing container" podUID="07b65b31-a557-4e4e-9dc8-589ddc9a9002" containerName="revoke-keys" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879453 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b65b31-a557-4e4e-9dc8-589ddc9a9002" containerName="revoke-keys" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879564 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b425e9fb-038d-4b47-859c-d5cf7c2c22dc" containerName="main" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879578 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1dd20c-2f12-4faf-9425-b2cf70938868" containerName="istio-proxy" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879587 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9c8fbb-ef28-4236-a453-343ce87b22bc" containerName="main" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879600 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0ccbd5-e95f-43bb-8811-961a812f2bc7" containerName="payload-pre-processing" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879610 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35d741f-a717-424d-a22e-5fd1b834e96f" containerName="payload-processing" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879620 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f8c12f-cbce-4387-a9e6-f3a5e22796ee" containerName="main" Nov 02 22:23:34.879717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.879628 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b65b31-a557-4e4e-9dc8-589ddc9a9002" containerName="revoke-keys" Nov 02 22:23:34.884805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.884784 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:34.989923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:34.989890 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss"] Nov 02 22:23:35.021403 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021571 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021419 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021571 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021508 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021743 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021637 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021743 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021699 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6l6\" (UniqueName: \"kubernetes.io/projected/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-kube-api-access-8w6l6\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021738 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021769 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021839 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.021917 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.021860 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.122773 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.122743 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.122911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.122775 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6l6\" (UniqueName: \"kubernetes.io/projected/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-kube-api-access-8w6l6\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.122911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.122807 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.122911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.122834 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.122911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.122876 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.122954 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.123068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123143 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.123104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123569 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.123171 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123868 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.123831 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.123885 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.123987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.123143 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.124329 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.124305 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.124454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.124322 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.126301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.126277 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.133406 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.133356 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.180169 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.180141 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.180766 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.180736 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6l6\" (UniqueName: \"kubernetes.io/projected/b7a3bf38-6dc8-4465-83ba-9eeebe01a791-kube-api-access-8w6l6\") pod \"maas-default-gateway-openshift-default-65c448b547-zw2ss\" (UID: \"b7a3bf38-6dc8-4465-83ba-9eeebe01a791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.198013 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.197989 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:35.619588 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.619559 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss"] Nov 02 22:23:35.621160 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:23:35.621131 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a3bf38_6dc8_4465_83ba_9eeebe01a791.slice/crio-0afd19cb9e0c4149e3edb7d1eaeb38bfcb01c0c140a0dee5dd4ec8ce1c11e619 WatchSource:0}: Error finding container 0afd19cb9e0c4149e3edb7d1eaeb38bfcb01c0c140a0dee5dd4ec8ce1c11e619: Status 404 returned error can't find the container with id 0afd19cb9e0c4149e3edb7d1eaeb38bfcb01c0c140a0dee5dd4ec8ce1c11e619 Nov 02 22:23:35.623374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.623322 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:35.623454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.623407 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:35.623454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:35.623437 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:36.192750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:36.192705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" event={"ID":"b7a3bf38-6dc8-4465-83ba-9eeebe01a791","Type":"ContainerStarted","Data":"022b1d6abe4d341d2c171b80b2fa927b0e5e1baeb07e9f226e3a5d2b2bbf6d06"} Nov 02 22:23:36.193160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:36.192766 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" event={"ID":"b7a3bf38-6dc8-4465-83ba-9eeebe01a791","Type":"ContainerStarted","Data":"0afd19cb9e0c4149e3edb7d1eaeb38bfcb01c0c140a0dee5dd4ec8ce1c11e619"} Nov 02 22:23:36.199138 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:36.199121 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:36.200798 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:36.200780 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:36.200863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:36.200822 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:36.310036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:36.309978 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podStartSLOduration=2.309961539 podStartE2EDuration="2.309961539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:36.308468421 +0000 UTC m=+2963.470563552" watchObservedRunningTime="2026-11-02 22:23:36.309961539 +0000 UTC m=+2963.472056659" Nov 02 22:23:37.114213 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:23:37.114176 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:23:37.198577 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:37.198537 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:37.199092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:37.198592 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:38.198409 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:38.198370 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:38.198627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:38.198433 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:39.198476 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:39.198442 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:39.198664 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:39.198491 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:40.198460 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:40.198422 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:40.198660 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:40.198507 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:41.044454 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:41.044419 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2"] Nov 02 22:23:41.094209 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:41.094171 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5416ece4-f9zj2"] Nov 02 22:23:41.199226 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:41.199193 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:41.199394 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:41.199247 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:42.198741 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:42.198702 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zw2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 22:23:42.199111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:42.198777 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" podUID="b7a3bf38-6dc8-4465-83ba-9eeebe01a791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 22:23:42.975685 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:42.975644 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6db464-1d01-42e0-8f73-76fa47919a9a" path="/var/lib/kubelet/pods/0c6db464-1d01-42e0-8f73-76fa47919a9a/volumes" Nov 02 22:23:43.791853 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:43.791820 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:43.792336 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:43.792257 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:43.793250 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:43.793218 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zw2ss" Nov 02 22:23:54.044596 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:54.044564 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt"] Nov 02 22:23:54.093883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:54.093849 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b8f318-2djwt"] Nov 02 22:23:54.976003 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:23:54.975959 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792e6a6f-be29-40e8-8152-41146a05af4e" path="/var/lib/kubelet/pods/792e6a6f-be29-40e8-8152-41146a05af4e/volumes" Nov 02 22:24:07.111927 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:24:07.111889 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:24:13.534864 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:24:13.534826 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:24:13.549149 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:24:13.549126 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:24:37.112565 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:24:37.112482 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:25:07.111680 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:25:07.111634 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:25:21.244778 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:21.244746 5091 scope.go:122] "RemoveContainer" containerID="86b04913865b33dce87844e894c63773775a6116faa6416f8bdedd2e39b646de" Nov 02 22:25:21.253939 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:21.253921 5091 scope.go:122] "RemoveContainer" containerID="dca994a0de5325d02dccf877e24d2d71bf8cbe555243b0239082977e0f59c175" Nov 02 22:25:36.043696 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:36.043630 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr"] Nov 02 22:25:36.093033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:36.092985 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b26a0d-rfcrr"] Nov 02 22:25:36.974606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:36.974573 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3187f782-a32a-4eb8-8165-8e331536d770" path="/var/lib/kubelet/pods/3187f782-a32a-4eb8-8165-8e331536d770/volumes" Nov 02 22:25:37.112129 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:25:37.112097 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:25:48.048287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:48.048251 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9"] Nov 02 22:25:48.097703 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:48.097663 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52049a7f-lwfg9"] Nov 02 22:25:48.974724 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:25:48.974690 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c230e278-7505-4fde-a5f5-e53561045821" path="/var/lib/kubelet/pods/c230e278-7505-4fde-a5f5-e53561045821/volumes" Nov 02 22:26:07.112461 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:26:07.112424 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:26:21.296694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:26:21.296622 5091 scope.go:122] "RemoveContainer" containerID="73caac335457d689642cf211893ca39d551e6002ee7490691be34b07090865f4" Nov 02 22:26:21.338610 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:26:21.338568 5091 scope.go:122] "RemoveContainer" containerID="58a956f83430f794c49a98469307ae26a4f0ecd75b6e259c5a8b07bec57ed16a" Nov 02 22:26:37.111852 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:26:37.111810 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:27:07.112087 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:27:07.112051 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:27:15.045374 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:15.045336 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j"] Nov 02 22:27:15.095063 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:15.095030 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037110d3-d4k6j"] Nov 02 22:27:16.977499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:16.977461 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04e2c12-0fa1-4b85-9d21-6b6c98388ff1" path="/var/lib/kubelet/pods/e04e2c12-0fa1-4b85-9d21-6b6c98388ff1/volumes" Nov 02 22:27:21.428061 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:21.428029 5091 scope.go:122] "RemoveContainer" containerID="637a1f72b9ab1e5cd838a8d7ccf4f5489915462f4164060fdea33d04f4305f03" Nov 02 22:27:21.437183 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:21.437161 5091 scope.go:122] "RemoveContainer" containerID="2382ebdd8597678f1aaa3a61bb8a6b1310e011d4e6606d68c27f4cc7473700d2" Nov 02 22:27:21.445350 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:21.445330 5091 scope.go:122] "RemoveContainer" containerID="c6ce02146c3aac076f5b8cb30f563d2f4edaf1701fc2287de48647e63c220c95" Nov 02 22:27:21.453221 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:21.453205 5091 scope.go:122] "RemoveContainer" containerID="9e36239df65a9bf2ea7855a87dd3d207f642869f8c42636586c6892623f1961b" Nov 02 22:27:21.460623 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:21.460607 5091 scope.go:122] "RemoveContainer" containerID="009dc33305d3b97d74192067d558462ddd6bda8aa4e164d1e3d91facfa754ace" Nov 02 22:27:30.051732 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:30.051623 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8"] Nov 02 22:27:30.100831 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:30.100800 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786b24e5-ts2r8"] Nov 02 22:27:30.975601 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:27:30.975568 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b65b31-a557-4e4e-9dc8-589ddc9a9002" path="/var/lib/kubelet/pods/07b65b31-a557-4e4e-9dc8-589ddc9a9002/volumes" Nov 02 22:27:37.111774 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:27:37.111739 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:28:07.111751 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:28:07.111715 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:28:21.494409 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:28:21.494373 5091 scope.go:122] "RemoveContainer" containerID="d94b14830cd3467bb03fce856ffcecfb2bf0b24376fca6b9fc27305498d23ca4" Nov 02 22:28:21.505710 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:28:21.505657 5091 scope.go:122] "RemoveContainer" containerID="19690018aee127a90de3fb7e71aa15bbac735bda196cc8b8b322a8300db60154" Nov 02 22:28:21.513861 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:28:21.513844 5091 scope.go:122] "RemoveContainer" containerID="540469f2af56566cdfe9981ac641a5bb7cf49f5169c1051fc25438ffdf36c20e" Nov 02 22:28:21.521568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:28:21.521550 5091 scope.go:122] "RemoveContainer" containerID="19acda02335e17187b6d5caad35f5eab97f0b827a928a0323d35f677536f9dde" Nov 02 22:28:37.111367 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:28:37.111326 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:29:07.112869 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:29:07.112832 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:29:13.568573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:29:13.568542 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:29:13.584397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:29:13.584376 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:29:21.551103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:29:21.551066 5091 scope.go:122] "RemoveContainer" containerID="eb6235ae91f3b312b72f4d5433f2e383c27038bc69096d7ef677ec431fa94302" Nov 02 22:29:37.111217 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:29:37.111182 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:30:00.154477 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.154435 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6"] Nov 02 22:30:00.158378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.158334 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.214575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.214540 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6"] Nov 02 22:30:00.233326 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.233294 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:30:00.261228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.261201 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:30:00.261228 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.261216 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:30:00.283266 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.283240 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vnb\" (UniqueName: \"kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb\") pod \"maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.283392 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.283289 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.384585 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.384548 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vnb\" (UniqueName: \"kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb\") pod \"maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.384795 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.384607 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.385294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.385274 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.441782 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.441715 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vnb\" (UniqueName: \"kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb\") pod \"maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.475497 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.475475 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:00.842282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:00.842206 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6"] Nov 02 22:30:00.844941 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:30:00.844912 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1be8a38_8a6f_42b7_9fd9_86a7623140d7.slice/crio-96885d31eacccde9860be106de5641e47f5b122a75c03c56ceb84ae1be57e39d WatchSource:0}: Error finding container 96885d31eacccde9860be106de5641e47f5b122a75c03c56ceb84ae1be57e39d: Status 404 returned error can't find the container with id 96885d31eacccde9860be106de5641e47f5b122a75c03c56ceb84ae1be57e39d Nov 02 22:30:01.442070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:01.441981 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerStarted","Data":"9ac7dba09f8688d9ef83fc5ecb038ec278c102cfde6a1bd06b50d85276bda3bb"} Nov 02 22:30:01.442070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:01.442033 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerStarted","Data":"96885d31eacccde9860be106de5641e47f5b122a75c03c56ceb84ae1be57e39d"} Nov 02 22:30:02.448231 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:02.448199 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerID="9ac7dba09f8688d9ef83fc5ecb038ec278c102cfde6a1bd06b50d85276bda3bb" exitCode=7 Nov 02 22:30:02.448652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:02.448270 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerDied","Data":"9ac7dba09f8688d9ef83fc5ecb038ec278c102cfde6a1bd06b50d85276bda3bb"} Nov 02 22:30:02.448652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:02.448466 5091 scope.go:122] "RemoveContainer" containerID="9ac7dba09f8688d9ef83fc5ecb038ec278c102cfde6a1bd06b50d85276bda3bb" Nov 02 22:30:03.454967 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:03.454936 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerID="b3bd6164a925de924759c8229b929bbd13ecb2be69ef820fccce942370ff1b33" exitCode=7 Nov 02 22:30:03.455378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:03.455020 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerDied","Data":"b3bd6164a925de924759c8229b929bbd13ecb2be69ef820fccce942370ff1b33"} Nov 02 22:30:03.455378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:03.455070 5091 scope.go:122] "RemoveContainer" containerID="9ac7dba09f8688d9ef83fc5ecb038ec278c102cfde6a1bd06b50d85276bda3bb" Nov 02 22:30:03.455378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:03.455305 5091 scope.go:122] "RemoveContainer" containerID="b3bd6164a925de924759c8229b929bbd13ecb2be69ef820fccce942370ff1b33" Nov 02 22:30:03.455598 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:30:03.455567 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6_odh-ai-gateway-infra(c1be8a38-8a6f-42b7-9fd9-86a7623140d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" Nov 02 22:30:04.462056 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:04.462031 5091 scope.go:122] "RemoveContainer" containerID="b3bd6164a925de924759c8229b929bbd13ecb2be69ef820fccce942370ff1b33" Nov 02 22:30:04.462464 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:30:04.462251 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6_odh-ai-gateway-infra(c1be8a38-8a6f-42b7-9fd9-86a7623140d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" Nov 02 22:30:07.111632 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:30:07.111594 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:30:16.969408 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:16.969375 5091 scope.go:122] "RemoveContainer" containerID="b3bd6164a925de924759c8229b929bbd13ecb2be69ef820fccce942370ff1b33" Nov 02 22:30:17.534385 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:17.534306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerStarted","Data":"36457ef7df85311fbde415100d37f030347114e4ac4bd9f4cb17398dcf2d5b0a"} Nov 02 22:30:17.648953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:17.648900 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" podStartSLOduration=17.648887163 podStartE2EDuration="17.648887163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:01.553250847 +0000 UTC m=+3348.715345966" watchObservedRunningTime="2026-11-02 22:30:17.648887163 +0000 UTC m=+3364.810982282" Nov 02 22:30:18.542465 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:18.542435 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerID="36457ef7df85311fbde415100d37f030347114e4ac4bd9f4cb17398dcf2d5b0a" exitCode=60 Nov 02 22:30:18.542913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:18.542472 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerDied","Data":"36457ef7df85311fbde415100d37f030347114e4ac4bd9f4cb17398dcf2d5b0a"} Nov 02 22:30:18.542913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:18.542518 5091 scope.go:122] "RemoveContainer" containerID="b3bd6164a925de924759c8229b929bbd13ecb2be69ef820fccce942370ff1b33" Nov 02 22:30:18.542913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:18.542857 5091 scope.go:122] "RemoveContainer" containerID="36457ef7df85311fbde415100d37f030347114e4ac4bd9f4cb17398dcf2d5b0a" Nov 02 22:30:18.543095 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:30:18.543072 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6_odh-ai-gateway-infra(c1be8a38-8a6f-42b7-9fd9-86a7623140d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" Nov 02 22:30:18.665268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:18.665236 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6"] Nov 02 22:30:19.681904 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.681878 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:19.750985 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.750957 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb\" (UniqueName: \"kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb\") pod \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " Nov 02 22:30:19.751131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.751012 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca\") pod \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\" (UID: \"c1be8a38-8a6f-42b7-9fd9-86a7623140d7\") " Nov 02 22:30:19.751387 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.751355 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca" pod "c1be8a38-8a6f-42b7-9fd9-86a7623140d7" (UID: "c1be8a38-8a6f-42b7-9fd9-86a7623140d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:19.753015 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.752991 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb" pod "c1be8a38-8a6f-42b7-9fd9-86a7623140d7" (UID: "c1be8a38-8a6f-42b7-9fd9-86a7623140d7"). InnerVolumeSpecName "kube-api-access-86vnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:19.852346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.852267 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vnb\" (UniqueName: \"kubernetes.io/projected/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-kube-api-access-86vnb\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:30:19.852346 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.852296 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be8a38-8a6f-42b7-9fd9-86a7623140d7-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:30:19.949778 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.949741 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q"] Nov 02 22:30:19.950170 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950152 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950172 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950200 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950208 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950244 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950253 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950387 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.950479 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.950409 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:30:19.953533 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:19.953514 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.007966 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.007935 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-53bfd6\"/\"kube-root-ca.crt\"" Nov 02 22:30:20.009033 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.008985 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-53bfd6\"/\"default-dockercfg-l27hs\"" Nov 02 22:30:20.009231 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.009207 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-53bfd6\"/\"openshift-service-ca.crt\"" Nov 02 22:30:20.009362 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.009342 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-53bfd6\"/\"simulator-single-main-53bfd6-kserve-self-signed-certs\"" Nov 02 22:30:20.011316 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.011294 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q"] Nov 02 22:30:20.054125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.054090 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.054294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.054142 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.054294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.054169 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.054294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.054271 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mkj\" (UniqueName: \"kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.054471 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.054455 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.054517 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.054504 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155137 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155105 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155144 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155195 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155218 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155238 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155351 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mkj\" (UniqueName: \"kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155713 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155684 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155782 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155699 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.155782 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.155763 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.157871 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.157845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.158036 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.158016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.212655 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.212623 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mkj\" (UniqueName: \"kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj\") pod \"simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.264813 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.264780 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:20.556373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.556299 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" Nov 02 22:30:20.556373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.556327 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6" event={"ID":"c1be8a38-8a6f-42b7-9fd9-86a7623140d7","Type":"ContainerDied","Data":"96885d31eacccde9860be106de5641e47f5b122a75c03c56ceb84ae1be57e39d"} Nov 02 22:30:20.556600 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.556395 5091 scope.go:122] "RemoveContainer" containerID="36457ef7df85311fbde415100d37f030347114e4ac4bd9f4cb17398dcf2d5b0a" Nov 02 22:30:20.639087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.639061 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q"] Nov 02 22:30:20.640637 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:30:20.640612 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a6446e4_fb46_4223_ae76_0a985b858b10.slice/crio-0fe7590471cb40568722f3951622e9fc21ad7010b6609cca5bdef5e14c62b47a WatchSource:0}: Error finding container 0fe7590471cb40568722f3951622e9fc21ad7010b6609cca5bdef5e14c62b47a: Status 404 returned error can't find the container with id 0fe7590471cb40568722f3951622e9fc21ad7010b6609cca5bdef5e14c62b47a Nov 02 22:30:20.720654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.720626 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6"] Nov 02 22:30:20.724785 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.724759 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-53bfd6-2989431plqc6"] Nov 02 22:30:20.975922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:20.975888 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" path="/var/lib/kubelet/pods/c1be8a38-8a6f-42b7-9fd9-86a7623140d7/volumes" Nov 02 22:30:21.564925 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:21.564889 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" event={"ID":"1a6446e4-fb46-4223-ae76-0a985b858b10","Type":"ContainerStarted","Data":"bc489e465c1af6311e6734162a39c1ee3f8ee7519a650b16ed7b3ad68570771e"} Nov 02 22:30:21.565118 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:21.564931 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" event={"ID":"1a6446e4-fb46-4223-ae76-0a985b858b10","Type":"ContainerStarted","Data":"0fe7590471cb40568722f3951622e9fc21ad7010b6609cca5bdef5e14c62b47a"} Nov 02 22:30:21.682654 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:21.682604 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" podStartSLOduration=2.682587535 podStartE2EDuration="2.682587535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:21.680131565 +0000 UTC m=+3368.842226685" watchObservedRunningTime="2026-11-02 22:30:21.682587535 +0000 UTC m=+3368.844682650" Nov 02 22:30:22.265378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:22.265345 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:30.265210 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:30.265133 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:30.277987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:30.277961 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:30.628788 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:30:30.628759 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:30:37.113071 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:30:37.113035 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:31:05.094327 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.094291 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c"] Nov 02 22:31:05.094825 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.094769 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1be8a38-8a6f-42b7-9fd9-86a7623140d7" containerName="cleanup" Nov 02 22:31:05.097879 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.097862 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.155140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.155114 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c"] Nov 02 22:31:05.163140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.163105 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.163304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.163170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lss\" (UniqueName: \"kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.163304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.163226 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.201695 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.201660 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:31:05.201836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.201694 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:31:05.201836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.201729 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:05.201836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.201799 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:31:05.263743 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.263713 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.263926 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.263758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lss\" (UniqueName: \"kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.263926 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.263779 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.264460 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.264441 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.266131 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.266100 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.319923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.319893 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lss\" (UniqueName: \"kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss\") pod \"maas-api-revoke-sub-212f0059-7gl6c\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.410222 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.410194 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:05.778333 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.778295 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c"] Nov 02 22:31:05.780128 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:31:05.780100 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5ab805_14ad_4a35_805f_ccb045853839.slice/crio-4b965072313ae161aba1e16a3ae3ae7eec278f581b16b09ba044238e7a609be5 WatchSource:0}: Error finding container 4b965072313ae161aba1e16a3ae3ae7eec278f581b16b09ba044238e7a609be5: Status 404 returned error can't find the container with id 4b965072313ae161aba1e16a3ae3ae7eec278f581b16b09ba044238e7a609be5 Nov 02 22:31:05.798632 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:05.798607 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" event={"ID":"0a5ab805-14ad-4a35-805f-ccb045853839","Type":"ContainerStarted","Data":"4b965072313ae161aba1e16a3ae3ae7eec278f581b16b09ba044238e7a609be5"} Nov 02 22:31:06.806206 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:06.806119 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a5ab805-14ad-4a35-805f-ccb045853839" containerID="5c0e663d1693ce8fff14faef9bfc9eeeb9d4cc380d48ebbc04285f8c7509c94e" exitCode=0 Nov 02 22:31:06.806592 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:06.806201 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" event={"ID":"0a5ab805-14ad-4a35-805f-ccb045853839","Type":"ContainerDied","Data":"5c0e663d1693ce8fff14faef9bfc9eeeb9d4cc380d48ebbc04285f8c7509c94e"} Nov 02 22:31:07.112091 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:31:07.112005 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:31:07.947218 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.947193 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:07.985591 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.985566 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca\") pod \"0a5ab805-14ad-4a35-805f-ccb045853839\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " Nov 02 22:31:07.985761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.985610 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss\" (UniqueName: \"kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss\") pod \"0a5ab805-14ad-4a35-805f-ccb045853839\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " Nov 02 22:31:07.985761 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.985640 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca\") pod \"0a5ab805-14ad-4a35-805f-ccb045853839\" (UID: \"0a5ab805-14ad-4a35-805f-ccb045853839\") " Nov 02 22:31:07.986012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.985983 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca" pod "0a5ab805-14ad-4a35-805f-ccb045853839" (UID: "0a5ab805-14ad-4a35-805f-ccb045853839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:07.987717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.987695 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss" pod "0a5ab805-14ad-4a35-805f-ccb045853839" (UID: "0a5ab805-14ad-4a35-805f-ccb045853839"). InnerVolumeSpecName "kube-api-access-j4lss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:07.987792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:07.987730 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca" pod "0a5ab805-14ad-4a35-805f-ccb045853839" (UID: "0a5ab805-14ad-4a35-805f-ccb045853839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:08.087252 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:08.087189 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:08.087252 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:08.087224 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lss\" (UniqueName: \"kubernetes.io/projected/0a5ab805-14ad-4a35-805f-ccb045853839-kube-api-access-j4lss\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:08.087252 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:08.087240 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5ab805-14ad-4a35-805f-ccb045853839-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:08.818092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:08.818059 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" Nov 02 22:31:08.818092 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:08.818069 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c" event={"ID":"0a5ab805-14ad-4a35-805f-ccb045853839","Type":"ContainerDied","Data":"4b965072313ae161aba1e16a3ae3ae7eec278f581b16b09ba044238e7a609be5"} Nov 02 22:31:08.818313 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:08.818118 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b965072313ae161aba1e16a3ae3ae7eec278f581b16b09ba044238e7a609be5" Nov 02 22:31:17.279493 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.279452 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92"] Nov 02 22:31:17.280122 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.280087 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a5ab805-14ad-4a35-805f-ccb045853839" containerName="revoke-keys" Nov 02 22:31:17.280122 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.280108 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5ab805-14ad-4a35-805f-ccb045853839" containerName="revoke-keys" Nov 02 22:31:17.280254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.280201 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5ab805-14ad-4a35-805f-ccb045853839" containerName="revoke-keys" Nov 02 22:31:17.283292 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.283270 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.369288 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.369254 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.369498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.369318 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.369498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.369413 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h479q\" (UniqueName: \"kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.386922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.386894 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:17.386922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.386903 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:31:17.387086 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.386938 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:31:17.387086 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.386956 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:31:17.388964 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.388941 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92"] Nov 02 22:31:17.470731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.470701 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.470870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.470748 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h479q\" (UniqueName: \"kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.470870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.470790 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.471336 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.471303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.473034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.473012 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.526806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.526776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h479q\" (UniqueName: \"kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.596034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.595962 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:17.964481 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:17.964455 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92"] Nov 02 22:31:17.966317 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:31:17.966276 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede72cd3_b0ec_4e55_adb8_217a6c1f9a6d.slice/crio-f96cbacab7eb79e2ba7aa145bdf4b67ae4f393131c5a4252bee0637d144d87b2 WatchSource:0}: Error finding container f96cbacab7eb79e2ba7aa145bdf4b67ae4f393131c5a4252bee0637d144d87b2: Status 404 returned error can't find the container with id f96cbacab7eb79e2ba7aa145bdf4b67ae4f393131c5a4252bee0637d144d87b2 Nov 02 22:31:18.868759 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:18.868727 5091 generic.go:356] "Generic (PLEG): container finished" podID="ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" containerID="a45aacb0911fd4302dd08fee703899ae67270543088da2d2a67aafe5b545caab" exitCode=0 Nov 02 22:31:18.869079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:18.868807 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" event={"ID":"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d","Type":"ContainerDied","Data":"a45aacb0911fd4302dd08fee703899ae67270543088da2d2a67aafe5b545caab"} Nov 02 22:31:18.869079 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:18.868857 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" event={"ID":"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d","Type":"ContainerStarted","Data":"f96cbacab7eb79e2ba7aa145bdf4b67ae4f393131c5a4252bee0637d144d87b2"} Nov 02 22:31:20.005172 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.005148 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:20.094918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.094887 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca\") pod \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " Nov 02 22:31:20.095054 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.094991 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q\" (UniqueName: \"kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q\") pod \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " Nov 02 22:31:20.095054 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.095038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca\") pod \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\" (UID: \"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d\") " Nov 02 22:31:20.095404 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.095379 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca" pod "ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" (UID: "ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:20.096883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.096863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca" pod "ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" (UID: "ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:20.096950 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.096925 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q" pod "ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" (UID: "ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d"). InnerVolumeSpecName "kube-api-access-h479q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:20.196135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.196053 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:20.196135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.196082 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:20.196135 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.196092 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h479q\" (UniqueName: \"kubernetes.io/projected/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d-kube-api-access-h479q\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:20.879833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.879801 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" Nov 02 22:31:20.879833 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.879816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92" event={"ID":"ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d","Type":"ContainerDied","Data":"f96cbacab7eb79e2ba7aa145bdf4b67ae4f393131c5a4252bee0637d144d87b2"} Nov 02 22:31:20.880110 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:20.879859 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f96cbacab7eb79e2ba7aa145bdf4b67ae4f393131c5a4252bee0637d144d87b2" Nov 02 22:31:21.771551 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:21.769173 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q"] Nov 02 22:31:35.664752 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:35.664714 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92"] Nov 02 22:31:35.713740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:35.713697 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-53bfd6-cnc92"] Nov 02 22:31:36.842193 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:36.842145 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" podUID="1a6446e4-fb46-4223-ae76-0a985b858b10" containerName="main" containerID="cri-o://bc489e465c1af6311e6734162a39c1ee3f8ee7519a650b16ed7b3ad68570771e" gracePeriod=45 Nov 02 22:31:36.961508 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:36.961482 5091 generic.go:356] "Generic (PLEG): container finished" podID="1a6446e4-fb46-4223-ae76-0a985b858b10" containerID="bc489e465c1af6311e6734162a39c1ee3f8ee7519a650b16ed7b3ad68570771e" exitCode=0 Nov 02 22:31:36.961617 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:36.961523 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" event={"ID":"1a6446e4-fb46-4223-ae76-0a985b858b10","Type":"ContainerDied","Data":"bc489e465c1af6311e6734162a39c1ee3f8ee7519a650b16ed7b3ad68570771e"} Nov 02 22:31:36.976911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:36.976884 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" path="/var/lib/kubelet/pods/ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d/volumes" Nov 02 22:31:37.091284 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.091259 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:31:37.114950 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:31:37.114879 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:31:37.150894 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.150866 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home\") pod \"1a6446e4-fb46-4223-ae76-0a985b858b10\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " Nov 02 22:31:37.151046 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.150909 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm\") pod \"1a6446e4-fb46-4223-ae76-0a985b858b10\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " Nov 02 22:31:37.151046 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.150927 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache\") pod \"1a6446e4-fb46-4223-ae76-0a985b858b10\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " Nov 02 22:31:37.151046 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.150977 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj\" (UniqueName: \"kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj\") pod \"1a6446e4-fb46-4223-ae76-0a985b858b10\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " Nov 02 22:31:37.151238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir\") pod \"1a6446e4-fb46-4223-ae76-0a985b858b10\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " Nov 02 22:31:37.151238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151089 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs\") pod \"1a6446e4-fb46-4223-ae76-0a985b858b10\" (UID: \"1a6446e4-fb46-4223-ae76-0a985b858b10\") " Nov 02 22:31:37.151238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151142 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home" pod "1a6446e4-fb46-4223-ae76-0a985b858b10" (UID: "1a6446e4-fb46-4223-ae76-0a985b858b10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:37.151238 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151223 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache" pod "1a6446e4-fb46-4223-ae76-0a985b858b10" (UID: "1a6446e4-fb46-4223-ae76-0a985b858b10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:37.151425 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151315 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir" pod "1a6446e4-fb46-4223-ae76-0a985b858b10" (UID: "1a6446e4-fb46-4223-ae76-0a985b858b10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:37.151461 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151437 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:37.151461 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151452 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:37.151581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.151461 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:37.153144 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.153122 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm" pod "1a6446e4-fb46-4223-ae76-0a985b858b10" (UID: "1a6446e4-fb46-4223-ae76-0a985b858b10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:37.153224 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.153189 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs" pod "1a6446e4-fb46-4223-ae76-0a985b858b10" (UID: "1a6446e4-fb46-4223-ae76-0a985b858b10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:37.153264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.153240 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj" pod "1a6446e4-fb46-4223-ae76-0a985b858b10" (UID: "1a6446e4-fb46-4223-ae76-0a985b858b10"). InnerVolumeSpecName "kube-api-access-m2mkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:37.252639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.252609 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6446e4-fb46-4223-ae76-0a985b858b10-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:37.252639 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.252637 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6446e4-fb46-4223-ae76-0a985b858b10-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:37.252873 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.252648 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mkj\" (UniqueName: \"kubernetes.io/projected/1a6446e4-fb46-4223-ae76-0a985b858b10-kube-api-access-m2mkj\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:37.969202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.969167 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" Nov 02 22:31:37.969202 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.969167 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q" event={"ID":"1a6446e4-fb46-4223-ae76-0a985b858b10","Type":"ContainerDied","Data":"0fe7590471cb40568722f3951622e9fc21ad7010b6609cca5bdef5e14c62b47a"} Nov 02 22:31:37.969800 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:37.969215 5091 scope.go:122] "RemoveContainer" containerID="bc489e465c1af6311e6734162a39c1ee3f8ee7519a650b16ed7b3ad68570771e" Nov 02 22:31:38.147220 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:38.147191 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q"] Nov 02 22:31:38.152445 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:38.152418 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-53bfd6/simulator-single-main-53bfd6-kserve-57596b5b6d-2fm6q"] Nov 02 22:31:38.976021 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:31:38.975978 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6446e4-fb46-4223-ae76-0a985b858b10" path="/var/lib/kubelet/pods/1a6446e4-fb46-4223-ae76-0a985b858b10/volumes" Nov 02 22:32:07.112680 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:32:07.112627 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:32:37.112332 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:32:37.112224 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:33:07.112550 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:33:07.112514 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:33:29.123815 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:33:29.123772 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:33:29.124511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:33:29.124478 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:33:37.112022 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:33:37.111983 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:34:07.112198 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:34:07.112160 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:34:13.611239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:13.611206 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:34:13.626821 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:13.626798 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:34:37.111891 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:34:37.111853 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:34:46.218976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.218941 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk"] Nov 02 22:34:46.219512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.219332 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1a6446e4-fb46-4223-ae76-0a985b858b10" containerName="main" Nov 02 22:34:46.219512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.219343 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6446e4-fb46-4223-ae76-0a985b858b10" containerName="main" Nov 02 22:34:46.219512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.219389 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" containerName="revoke-keys" Nov 02 22:34:46.219512 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.219396 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" containerName="revoke-keys" Nov 02 22:34:46.219717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.219539 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede72cd3-b0ec-4e55-adb8-217a6c1f9a6d" containerName="revoke-keys" Nov 02 22:34:46.219717 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.219556 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6446e4-fb46-4223-ae76-0a985b858b10" containerName="main" Nov 02 22:34:46.222829 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.222810 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.274886 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.274856 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8b2f0b\"/\"kube-root-ca.crt\"" Nov 02 22:34:46.274886 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.274874 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8b2f0b\"/\"openshift-service-ca.crt\"" Nov 02 22:34:46.324716 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.324642 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8b2f0b\"/\"simulator-single-main-8b2f0b-kserve-self-signed-certs\"" Nov 02 22:34:46.324716 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.324687 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8b2f0b\"/\"default-dockercfg-jzblb\"" Nov 02 22:34:46.326726 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.326704 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk"] Nov 02 22:34:46.338282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.338259 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.338362 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.338290 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.338362 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.338314 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpcm\" (UniqueName: \"kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.338441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.338388 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.338441 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.338427 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.338560 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.338456 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.438931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.438897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.438931 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.438934 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpcm\" (UniqueName: \"kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.438965 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.439011 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.439046 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.439092 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.439319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439451 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.439390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.439566 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.439475 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.441343 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.441316 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.441699 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.441658 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.495297 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.495235 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpcm\" (UniqueName: \"kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm\") pod \"simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.534420 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.534393 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:46.910932 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.910903 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk"] Nov 02 22:34:46.912598 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:34:46.912567 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3113eaa_5330_4db3_a920_1ca30343d85c.slice/crio-f6abf1c703bf6c0d618a190968f5e71c31cade52b692afa905d7335e672f44cf WatchSource:0}: Error finding container f6abf1c703bf6c0d618a190968f5e71c31cade52b692afa905d7335e672f44cf: Status 404 returned error can't find the container with id f6abf1c703bf6c0d618a190968f5e71c31cade52b692afa905d7335e672f44cf Nov 02 22:34:46.975375 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.975339 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" event={"ID":"e3113eaa-5330-4db3-a920-1ca30343d85c","Type":"ContainerStarted","Data":"b3f8871541313e1fbc00b748208f44491ece65b758c030df0dc297b3f3613b63"} Nov 02 22:34:46.975483 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:46.975384 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" event={"ID":"e3113eaa-5330-4db3-a920-1ca30343d85c","Type":"ContainerStarted","Data":"f6abf1c703bf6c0d618a190968f5e71c31cade52b692afa905d7335e672f44cf"} Nov 02 22:34:47.089880 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:47.089821 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" podStartSLOduration=1.089807918 podStartE2EDuration="1.089807918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:47.089644843 +0000 UTC m=+3634.251739964" watchObservedRunningTime="2026-11-02 22:34:47.089807918 +0000 UTC m=+3634.251903038" Nov 02 22:34:47.534816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:47.534785 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:56.537988 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:56.537885 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:56.550434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:56.550408 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:34:57.033805 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:34:57.033776 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:35:07.113544 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:35:07.113507 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:35:37.111906 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:35:37.111873 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:35:47.103239 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:35:47.103201 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk"] Nov 02 22:36:02.123044 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.122990 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" podUID="e3113eaa-5330-4db3-a920-1ca30343d85c" containerName="main" containerID="cri-o://b3f8871541313e1fbc00b748208f44491ece65b758c030df0dc297b3f3613b63" gracePeriod=45 Nov 02 22:36:02.368869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.368841 5091 generic.go:356] "Generic (PLEG): container finished" podID="e3113eaa-5330-4db3-a920-1ca30343d85c" containerID="b3f8871541313e1fbc00b748208f44491ece65b758c030df0dc297b3f3613b63" exitCode=0 Nov 02 22:36:02.369030 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.368921 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" event={"ID":"e3113eaa-5330-4db3-a920-1ca30343d85c","Type":"ContainerDied","Data":"b3f8871541313e1fbc00b748208f44491ece65b758c030df0dc297b3f3613b63"} Nov 02 22:36:02.369030 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.368962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" event={"ID":"e3113eaa-5330-4db3-a920-1ca30343d85c","Type":"ContainerDied","Data":"f6abf1c703bf6c0d618a190968f5e71c31cade52b692afa905d7335e672f44cf"} Nov 02 22:36:02.369030 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.368973 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6abf1c703bf6c0d618a190968f5e71c31cade52b692afa905d7335e672f44cf" Nov 02 22:36:02.373012 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.372992 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:36:02.519294 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.519203 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm\") pod \"e3113eaa-5330-4db3-a920-1ca30343d85c\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " Nov 02 22:36:02.519474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.519288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs\") pod \"e3113eaa-5330-4db3-a920-1ca30343d85c\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " Nov 02 22:36:02.519474 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.519364 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm\" (UniqueName: \"kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm\") pod \"e3113eaa-5330-4db3-a920-1ca30343d85c\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " Nov 02 22:36:02.519613 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.519505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir\") pod \"e3113eaa-5330-4db3-a920-1ca30343d85c\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " Nov 02 22:36:02.519697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.519612 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home\") pod \"e3113eaa-5330-4db3-a920-1ca30343d85c\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " Nov 02 22:36:02.519697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.519653 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache\") pod \"e3113eaa-5330-4db3-a920-1ca30343d85c\" (UID: \"e3113eaa-5330-4db3-a920-1ca30343d85c\") " Nov 02 22:36:02.520627 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.520595 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir" pod "e3113eaa-5330-4db3-a920-1ca30343d85c" (UID: "e3113eaa-5330-4db3-a920-1ca30343d85c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:02.520783 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.520686 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home" pod "e3113eaa-5330-4db3-a920-1ca30343d85c" (UID: "e3113eaa-5330-4db3-a920-1ca30343d85c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:02.520783 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.520743 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache" pod "e3113eaa-5330-4db3-a920-1ca30343d85c" (UID: "e3113eaa-5330-4db3-a920-1ca30343d85c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:02.521853 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.521807 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm" pod "e3113eaa-5330-4db3-a920-1ca30343d85c" (UID: "e3113eaa-5330-4db3-a920-1ca30343d85c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:02.521976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.521916 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs" pod "e3113eaa-5330-4db3-a920-1ca30343d85c" (UID: "e3113eaa-5330-4db3-a920-1ca30343d85c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:02.521976 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.521945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm" pod "e3113eaa-5330-4db3-a920-1ca30343d85c" (UID: "e3113eaa-5330-4db3-a920-1ca30343d85c"). InnerVolumeSpecName "kube-api-access-tfpcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:02.620806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.620769 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:02.620806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.620801 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:02.620806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.620813 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:02.621040 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.620826 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3113eaa-5330-4db3-a920-1ca30343d85c-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:02.621040 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.620840 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfpcm\" (UniqueName: \"kubernetes.io/projected/e3113eaa-5330-4db3-a920-1ca30343d85c-kube-api-access-tfpcm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:02.621040 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:02.620853 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3113eaa-5330-4db3-a920-1ca30343d85c-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:03.374436 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:03.374398 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk" Nov 02 22:36:03.542160 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:03.542128 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk"] Nov 02 22:36:03.591884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:03.591853 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8b2f0b/simulator-single-main-8b2f0b-kserve-65f8f7b67d-wvcbk"] Nov 02 22:36:04.975951 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:04.975915 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3113eaa-5330-4db3-a920-1ca30343d85c" path="/var/lib/kubelet/pods/e3113eaa-5330-4db3-a920-1ca30343d85c/volumes" Nov 02 22:36:07.110956 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:36:07.110923 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:36:09.044854 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:09.044819 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c"] Nov 02 22:36:09.096732 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:09.096697 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212f0059-7gl6c"] Nov 02 22:36:10.975083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:10.975048 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5ab805-14ad-4a35-805f-ccb045853839" path="/var/lib/kubelet/pods/0a5ab805-14ad-4a35-805f-ccb045853839/volumes" Nov 02 22:36:15.005854 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.005818 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z"] Nov 02 22:36:15.006256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.006196 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e3113eaa-5330-4db3-a920-1ca30343d85c" containerName="main" Nov 02 22:36:15.006256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.006207 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3113eaa-5330-4db3-a920-1ca30343d85c" containerName="main" Nov 02 22:36:15.006337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.006284 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3113eaa-5330-4db3-a920-1ca30343d85c" containerName="main" Nov 02 22:36:15.010646 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.010625 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.031863 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.031834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.031977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.031869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jsx\" (UniqueName: \"kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.031977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.031919 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.032102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.032083 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.032140 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.032125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.032206 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.032190 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.066133 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.066103 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z"] Nov 02 22:36:15.113705 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.113663 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:36:15.113881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.113730 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:36:15.115265 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.115247 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jr9p4\"" Nov 02 22:36:15.115314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.115276 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-54f3fe42-kserve-self-signed-certs\"" Nov 02 22:36:15.133323 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133298 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133447 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133325 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133447 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133358 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133447 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133416 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133447 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133439 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jsx\" (UniqueName: \"kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133701 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133456 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133809 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133787 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133881 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133830 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.133926 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.133878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.135663 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.135631 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.136006 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.135989 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.193694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.193651 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jsx\" (UniqueName: \"kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx\") pod \"e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.322998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.322909 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:15.707270 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:15.707239 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z"] Nov 02 22:36:15.708124 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:36:15.708101 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5057ae13_1ed2_4792_b18a_8d7236e559fb.slice/crio-12b4c05c7ef6466fd8405996840ad5bfe7b5b99bd49a62538ae55f1f9947ef3c WatchSource:0}: Error finding container 12b4c05c7ef6466fd8405996840ad5bfe7b5b99bd49a62538ae55f1f9947ef3c: Status 404 returned error can't find the container with id 12b4c05c7ef6466fd8405996840ad5bfe7b5b99bd49a62538ae55f1f9947ef3c Nov 02 22:36:16.441993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:16.441957 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" event={"ID":"5057ae13-1ed2-4792-b18a-8d7236e559fb","Type":"ContainerStarted","Data":"a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9"} Nov 02 22:36:16.441993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:16.441995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" event={"ID":"5057ae13-1ed2-4792-b18a-8d7236e559fb","Type":"ContainerStarted","Data":"12b4c05c7ef6466fd8405996840ad5bfe7b5b99bd49a62538ae55f1f9947ef3c"} Nov 02 22:36:16.555600 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:16.555541 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" podStartSLOduration=2.555527413 podStartE2EDuration="2.555527413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:16.555110404 +0000 UTC m=+3723.717205526" watchObservedRunningTime="2026-11-02 22:36:16.555527413 +0000 UTC m=+3723.717622532" Nov 02 22:36:17.324041 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:17.324007 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:25.323650 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:25.323566 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:25.335830 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:25.335806 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:25.533285 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:25.533249 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:36:37.111514 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:36:37.111475 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:36:57.416121 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.416084 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz"] Nov 02 22:36:57.419470 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.419451 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.521225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.521191 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-54f3fe42-kserve-self-signed-certs\"" Nov 02 22:36:57.523270 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.523244 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz"] Nov 02 22:36:57.564787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.564749 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwpg\" (UniqueName: \"kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.564941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.564822 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.564941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.564910 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.565035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.564964 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.565035 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.564990 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.565145 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.565126 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.665943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.665907 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.665943 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.665944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.665972 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.665987 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.666014 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666232 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.666102 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwpg\" (UniqueName: \"kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666462 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.666329 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666462 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.666448 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.666535 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.666489 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.668268 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.668250 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.668659 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.668639 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.723568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.723541 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwpg\" (UniqueName: \"kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg\") pod \"e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:57.731364 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:57.731340 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:58.109240 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:58.109212 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz"] Nov 02 22:36:58.112001 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:36:58.111969 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06d57791_8956_4ae7_8d22_5db449a8a0a0.slice/crio-8ad4750d151474fb7b25a6a71aeeb4a1a3e98885b85d79b393c314b1185921ef WatchSource:0}: Error finding container 8ad4750d151474fb7b25a6a71aeeb4a1a3e98885b85d79b393c314b1185921ef: Status 404 returned error can't find the container with id 8ad4750d151474fb7b25a6a71aeeb4a1a3e98885b85d79b393c314b1185921ef Nov 02 22:36:58.691852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:58.691816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" event={"ID":"06d57791-8956-4ae7-8d22-5db449a8a0a0","Type":"ContainerStarted","Data":"00f100fa9818124a6890356a637fd472994354e9b534c3d29a893a720e688dc7"} Nov 02 22:36:58.692216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:58.691858 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" event={"ID":"06d57791-8956-4ae7-8d22-5db449a8a0a0","Type":"ContainerStarted","Data":"8ad4750d151474fb7b25a6a71aeeb4a1a3e98885b85d79b393c314b1185921ef"} Nov 02 22:36:58.732304 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:58.732280 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:36:58.807355 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:36:58.807301 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" podStartSLOduration=1.8072868949999998 podStartE2EDuration="1.807286895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:58.806037057 +0000 UTC m=+3765.968132176" watchObservedRunningTime="2026-11-02 22:36:58.807286895 +0000 UTC m=+3765.969382015" Nov 02 22:37:07.111304 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:37:07.111267 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:37:07.731890 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:07.731853 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:37:07.744628 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:07.744602 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:37:07.764583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:07.764553 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:37:21.653070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:21.653038 5091 scope.go:122] "RemoveContainer" containerID="5c0e663d1693ce8fff14faef9bfc9eeeb9d4cc380d48ebbc04285f8c7509c94e" Nov 02 22:37:21.662287 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:21.662260 5091 scope.go:122] "RemoveContainer" containerID="a45aacb0911fd4302dd08fee703899ae67270543088da2d2a67aafe5b545caab" Nov 02 22:37:37.112696 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:37:37.112584 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:37:44.414867 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:44.414832 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z"] Nov 02 22:37:54.918411 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:54.918324 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz"] Nov 02 22:37:59.434684 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.434628 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" podUID="5057ae13-1ed2-4792-b18a-8d7236e559fb" containerName="main" containerID="cri-o://a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9" gracePeriod=45 Nov 02 22:37:59.522816 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.522776 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" podUID="5057ae13-1ed2-4792-b18a-8d7236e559fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.250:8000/ready\": dial tcp 10.132.0.250:8000: connect: connection refused" Nov 02 22:37:59.691330 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.691273 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:37:59.804878 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.804839 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir\") pod \"5057ae13-1ed2-4792-b18a-8d7236e559fb\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " Nov 02 22:37:59.805069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.804900 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx\" (UniqueName: \"kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx\") pod \"5057ae13-1ed2-4792-b18a-8d7236e559fb\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " Nov 02 22:37:59.805069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.804993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache\") pod \"5057ae13-1ed2-4792-b18a-8d7236e559fb\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " Nov 02 22:37:59.805069 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.805043 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home\") pod \"5057ae13-1ed2-4792-b18a-8d7236e559fb\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " Nov 02 22:37:59.805251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.805086 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs\") pod \"5057ae13-1ed2-4792-b18a-8d7236e559fb\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " Nov 02 22:37:59.805251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.805119 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm\") pod \"5057ae13-1ed2-4792-b18a-8d7236e559fb\" (UID: \"5057ae13-1ed2-4792-b18a-8d7236e559fb\") " Nov 02 22:37:59.806348 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.806315 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home" pod "5057ae13-1ed2-4792-b18a-8d7236e559fb" (UID: "5057ae13-1ed2-4792-b18a-8d7236e559fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:37:59.806534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.806399 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir" pod "5057ae13-1ed2-4792-b18a-8d7236e559fb" (UID: "5057ae13-1ed2-4792-b18a-8d7236e559fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:37:59.806534 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.806458 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache" pod "5057ae13-1ed2-4792-b18a-8d7236e559fb" (UID: "5057ae13-1ed2-4792-b18a-8d7236e559fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:37:59.810173 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.810127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx" pod "5057ae13-1ed2-4792-b18a-8d7236e559fb" (UID: "5057ae13-1ed2-4792-b18a-8d7236e559fb"). InnerVolumeSpecName "kube-api-access-p9jsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:37:59.810562 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.810509 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs" pod "5057ae13-1ed2-4792-b18a-8d7236e559fb" (UID: "5057ae13-1ed2-4792-b18a-8d7236e559fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:37:59.812165 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.812137 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm" pod "5057ae13-1ed2-4792-b18a-8d7236e559fb" (UID: "5057ae13-1ed2-4792-b18a-8d7236e559fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:37:59.906230 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.906185 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:59.906230 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.906229 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jsx\" (UniqueName: \"kubernetes.io/projected/5057ae13-1ed2-4792-b18a-8d7236e559fb-kube-api-access-p9jsx\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:59.906515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.906248 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:59.906515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.906263 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:59.906515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.906277 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5057ae13-1ed2-4792-b18a-8d7236e559fb-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:59.906515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:37:59.906290 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5057ae13-1ed2-4792-b18a-8d7236e559fb-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:00.083781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.083692 5091 generic.go:356] "Generic (PLEG): container finished" podID="5057ae13-1ed2-4792-b18a-8d7236e559fb" containerID="a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9" exitCode=0 Nov 02 22:38:00.083781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.083709 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" event={"ID":"5057ae13-1ed2-4792-b18a-8d7236e559fb","Type":"ContainerDied","Data":"a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9"} Nov 02 22:38:00.083781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.083737 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" Nov 02 22:38:00.083781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.083772 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z" event={"ID":"5057ae13-1ed2-4792-b18a-8d7236e559fb","Type":"ContainerDied","Data":"12b4c05c7ef6466fd8405996840ad5bfe7b5b99bd49a62538ae55f1f9947ef3c"} Nov 02 22:38:00.084180 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.083799 5091 scope.go:122] "RemoveContainer" containerID="a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9" Nov 02 22:38:00.100016 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.099995 5091 scope.go:122] "RemoveContainer" containerID="a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9" Nov 02 22:38:00.100329 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:38:00.100310 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9\": container with ID starting with a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9 not found: ID does not exist" containerID="a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9" Nov 02 22:38:00.100395 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.100341 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9"} err="failed to get container status \"a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9\": rpc error: code = NotFound desc = could not find container \"a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9\": container with ID starting with a3101f7592dc970613167e696a706f07b873a35b27abd33735a94d1fd43bcdc9 not found: ID does not exist" Nov 02 22:38:00.246841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.246797 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z"] Nov 02 22:38:00.248754 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:00.248723 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-54f3fe42-kserve-566bb87c69-m2g9z"] Nov 02 22:38:01.016706 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:01.014331 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5057ae13-1ed2-4792-b18a-8d7236e559fb" path="/var/lib/kubelet/pods/5057ae13-1ed2-4792-b18a-8d7236e559fb/volumes" Nov 02 22:38:07.112226 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:38:07.112182 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:38:09.938129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:09.938073 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" podUID="06d57791-8956-4ae7-8d22-5db449a8a0a0" containerName="main" containerID="cri-o://00f100fa9818124a6890356a637fd472994354e9b534c3d29a893a720e688dc7" gracePeriod=45 Nov 02 22:38:10.166148 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.166115 5091 generic.go:356] "Generic (PLEG): container finished" podID="06d57791-8956-4ae7-8d22-5db449a8a0a0" containerID="00f100fa9818124a6890356a637fd472994354e9b534c3d29a893a720e688dc7" exitCode=0 Nov 02 22:38:10.166307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.166196 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" event={"ID":"06d57791-8956-4ae7-8d22-5db449a8a0a0","Type":"ContainerDied","Data":"00f100fa9818124a6890356a637fd472994354e9b534c3d29a893a720e688dc7"} Nov 02 22:38:10.195877 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.195820 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:38:10.317923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.317899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs\") pod \"06d57791-8956-4ae7-8d22-5db449a8a0a0\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " Nov 02 22:38:10.318058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.317949 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm\") pod \"06d57791-8956-4ae7-8d22-5db449a8a0a0\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " Nov 02 22:38:10.318058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.317974 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir\") pod \"06d57791-8956-4ae7-8d22-5db449a8a0a0\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " Nov 02 22:38:10.318058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.318013 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache\") pod \"06d57791-8956-4ae7-8d22-5db449a8a0a0\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " Nov 02 22:38:10.318058 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.318045 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg\" (UniqueName: \"kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg\") pod \"06d57791-8956-4ae7-8d22-5db449a8a0a0\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " Nov 02 22:38:10.318295 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.318085 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home\") pod \"06d57791-8956-4ae7-8d22-5db449a8a0a0\" (UID: \"06d57791-8956-4ae7-8d22-5db449a8a0a0\") " Nov 02 22:38:10.318412 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.318296 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir" pod "06d57791-8956-4ae7-8d22-5db449a8a0a0" (UID: "06d57791-8956-4ae7-8d22-5db449a8a0a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:38:10.318412 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.318334 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache" pod "06d57791-8956-4ae7-8d22-5db449a8a0a0" (UID: "06d57791-8956-4ae7-8d22-5db449a8a0a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:38:10.318521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.318491 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home" pod "06d57791-8956-4ae7-8d22-5db449a8a0a0" (UID: "06d57791-8956-4ae7-8d22-5db449a8a0a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:38:10.320083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.320050 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm" pod "06d57791-8956-4ae7-8d22-5db449a8a0a0" (UID: "06d57791-8956-4ae7-8d22-5db449a8a0a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:38:10.320204 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.320183 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs" pod "06d57791-8956-4ae7-8d22-5db449a8a0a0" (UID: "06d57791-8956-4ae7-8d22-5db449a8a0a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:10.320303 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.320276 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg" pod "06d57791-8956-4ae7-8d22-5db449a8a0a0" (UID: "06d57791-8956-4ae7-8d22-5db449a8a0a0"). InnerVolumeSpecName "kube-api-access-pfwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:10.418938 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.418904 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-dshm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:10.419099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.418953 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-tmp-dir\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:10.419099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.418967 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-model-cache\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:10.419099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.418980 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwpg\" (UniqueName: \"kubernetes.io/projected/06d57791-8956-4ae7-8d22-5db449a8a0a0-kube-api-access-pfwpg\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:10.419099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.418993 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d57791-8956-4ae7-8d22-5db449a8a0a0-home\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:10.419099 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:10.419005 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d57791-8956-4ae7-8d22-5db449a8a0a0-tls-certs\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:11.173216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:11.173183 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" Nov 02 22:38:11.173216 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:11.173198 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz" event={"ID":"06d57791-8956-4ae7-8d22-5db449a8a0a0","Type":"ContainerDied","Data":"8ad4750d151474fb7b25a6a71aeeb4a1a3e98885b85d79b393c314b1185921ef"} Nov 02 22:38:11.173856 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:11.173245 5091 scope.go:122] "RemoveContainer" containerID="00f100fa9818124a6890356a637fd472994354e9b534c3d29a893a720e688dc7" Nov 02 22:38:11.337941 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:11.337867 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz"] Nov 02 22:38:11.343644 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:11.343616 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-54f3fe42-kserve-8658b8dbf-45sfz"] Nov 02 22:38:13.003694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:12.999328 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d57791-8956-4ae7-8d22-5db449a8a0a0" path="/var/lib/kubelet/pods/06d57791-8956-4ae7-8d22-5db449a8a0a0/volumes" Nov 02 22:38:13.877693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:13.875027 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:38:13.877693 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:38:13.875092 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:38:37.111499 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:38:37.111468 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:39:07.112020 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:39:07.111987 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:39:13.649977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:13.649944 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:39:13.663579 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:13.663556 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:39:29.768197 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768158 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24"] Nov 02 22:39:29.768776 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768747 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5057ae13-1ed2-4792-b18a-8d7236e559fb" containerName="main" Nov 02 22:39:29.768949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768778 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5057ae13-1ed2-4792-b18a-8d7236e559fb" containerName="main" Nov 02 22:39:29.768949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768814 5091 cpu_manager.go:414] "containerMap: removing container" podUID="06d57791-8956-4ae7-8d22-5db449a8a0a0" containerName="main" Nov 02 22:39:29.768949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768822 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d57791-8956-4ae7-8d22-5db449a8a0a0" containerName="main" Nov 02 22:39:29.768949 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768940 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5057ae13-1ed2-4792-b18a-8d7236e559fb" containerName="main" Nov 02 22:39:29.769180 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.768956 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d57791-8956-4ae7-8d22-5db449a8a0a0" containerName="main" Nov 02 22:39:29.772187 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.772164 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.826367 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.826333 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:39:29.828983 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.828955 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24"] Nov 02 22:39:29.842979 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.842955 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.843097 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.842986 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.843097 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.843010 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9w9\" (UniqueName: \"kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.875866 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.875837 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:39:29.875989 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.875870 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:39:29.876047 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.875993 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zcnzx\"" Nov 02 22:39:29.943563 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.943537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.943731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.943570 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.943731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.943591 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9w9\" (UniqueName: \"kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.944264 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.944242 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.946152 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.946129 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:29.999650 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:29.999620 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9w9\" (UniqueName: \"kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9\") pod \"maas-api-revoke-keys-e2e-ait-ce65d173-m5p24\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:30.085045 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:30.084974 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:30.454551 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:30.454524 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24"] Nov 02 22:39:30.456537 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:39:30.456508 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f06bafe_b8c6_448e_9705_c982bd008eec.slice/crio-7ab889f7a5db6ea0f5fabd4c0dd7fd95f375151401ed3a6cbad0a7ff0f971b96 WatchSource:0}: Error finding container 7ab889f7a5db6ea0f5fabd4c0dd7fd95f375151401ed3a6cbad0a7ff0f971b96: Status 404 returned error can't find the container with id 7ab889f7a5db6ea0f5fabd4c0dd7fd95f375151401ed3a6cbad0a7ff0f971b96 Nov 02 22:39:30.605970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:30.605927 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" event={"ID":"9f06bafe-b8c6-448e-9705-c982bd008eec","Type":"ContainerStarted","Data":"7ab889f7a5db6ea0f5fabd4c0dd7fd95f375151401ed3a6cbad0a7ff0f971b96"} Nov 02 22:39:31.612397 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:31.612311 5091 generic.go:356] "Generic (PLEG): container finished" podID="9f06bafe-b8c6-448e-9705-c982bd008eec" containerID="e054394caed6f70690a5f5bc4644a3fbf23f35da764210e93ba088f69dc483b7" exitCode=0 Nov 02 22:39:31.612792 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:31.612397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" event={"ID":"9f06bafe-b8c6-448e-9705-c982bd008eec","Type":"ContainerDied","Data":"e054394caed6f70690a5f5bc4644a3fbf23f35da764210e93ba088f69dc483b7"} Nov 02 22:39:32.753103 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.753077 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:32.868244 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.868210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca\") pod \"9f06bafe-b8c6-448e-9705-c982bd008eec\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " Nov 02 22:39:32.868426 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.868296 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9\" (UniqueName: \"kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9\") pod \"9f06bafe-b8c6-448e-9705-c982bd008eec\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " Nov 02 22:39:32.868426 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.868336 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca\") pod \"9f06bafe-b8c6-448e-9705-c982bd008eec\" (UID: \"9f06bafe-b8c6-448e-9705-c982bd008eec\") " Nov 02 22:39:32.868808 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.868777 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca" pod "9f06bafe-b8c6-448e-9705-c982bd008eec" (UID: "9f06bafe-b8c6-448e-9705-c982bd008eec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:32.870828 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.870803 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca" pod "9f06bafe-b8c6-448e-9705-c982bd008eec" (UID: "9f06bafe-b8c6-448e-9705-c982bd008eec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:39:32.870913 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.870837 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9" pod "9f06bafe-b8c6-448e-9705-c982bd008eec" (UID: "9f06bafe-b8c6-448e-9705-c982bd008eec"). InnerVolumeSpecName "kube-api-access-ml9w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:32.969953 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.969921 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:32.970070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.969957 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9w9\" (UniqueName: \"kubernetes.io/projected/9f06bafe-b8c6-448e-9705-c982bd008eec-kube-api-access-ml9w9\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:32.970070 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:32.969974 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f06bafe-b8c6-448e-9705-c982bd008eec-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:33.627516 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:33.627484 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" Nov 02 22:39:33.627747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:33.627518 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24" event={"ID":"9f06bafe-b8c6-448e-9705-c982bd008eec","Type":"ContainerDied","Data":"7ab889f7a5db6ea0f5fabd4c0dd7fd95f375151401ed3a6cbad0a7ff0f971b96"} Nov 02 22:39:33.627747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:33.627566 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab889f7a5db6ea0f5fabd4c0dd7fd95f375151401ed3a6cbad0a7ff0f971b96" Nov 02 22:39:37.112443 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:39:37.112407 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:39:48.763834 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:48.763794 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24"] Nov 02 22:39:48.813261 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:48.813229 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce65d173-m5p24"] Nov 02 22:39:48.976977 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:39:48.976935 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f06bafe-b8c6-448e-9705-c982bd008eec" path="/var/lib/kubelet/pods/9f06bafe-b8c6-448e-9705-c982bd008eec/volumes" Nov 02 22:40:07.111875 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:40:07.111840 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:40:37.111714 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:40:37.111661 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:41:07.114451 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:41:07.114370 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:41:21.719432 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:21.719398 5091 scope.go:122] "RemoveContainer" containerID="b3f8871541313e1fbc00b748208f44491ece65b758c030df0dc297b3f3613b63" Nov 02 22:41:37.111922 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:41:37.111876 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:41:44.205869 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.205832 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg"] Nov 02 22:41:44.206359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.206230 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9f06bafe-b8c6-448e-9705-c982bd008eec" containerName="revoke-keys" Nov 02 22:41:44.206359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.206243 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f06bafe-b8c6-448e-9705-c982bd008eec" containerName="revoke-keys" Nov 02 22:41:44.206359 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.206304 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f06bafe-b8c6-448e-9705-c982bd008eec" containerName="revoke-keys" Nov 02 22:41:44.210806 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.210789 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.264735 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.264705 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:41:44.314052 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.314023 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:41:44.314219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.314060 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:41:44.314219 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.314073 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zcnzx\"" Nov 02 22:41:44.315934 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.315914 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg"] Nov 02 22:41:44.320362 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.320345 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9gd\" (UniqueName: \"kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.320434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.320368 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.320434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.320387 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.421841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.421813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9gd\" (UniqueName: \"kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.421841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.421842 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.422032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.421860 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.422442 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.422423 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.424156 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.424133 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.478503 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.478444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9gd\" (UniqueName: \"kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.522979 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.522956 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:44.891281 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:44.891254 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg"] Nov 02 22:41:44.893541 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:41:44.893514 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f2d9f3_0a30_4857_83ad_a720b344ab35.slice/crio-08fad185e03493d30dc17b9e2b71b635dfc321f1e693caf1bc1cb54f2736562f WatchSource:0}: Error finding container 08fad185e03493d30dc17b9e2b71b635dfc321f1e693caf1bc1cb54f2736562f: Status 404 returned error can't find the container with id 08fad185e03493d30dc17b9e2b71b635dfc321f1e693caf1bc1cb54f2736562f Nov 02 22:41:45.287324 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:45.287286 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" event={"ID":"79f2d9f3-0a30-4857-83ad-a720b344ab35","Type":"ContainerStarted","Data":"f405db3bcece2ac2fd7bc1ed86edb277ab6db81f6387ddce0e6b75eed42a5c20"} Nov 02 22:41:45.287841 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:45.287347 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" event={"ID":"79f2d9f3-0a30-4857-83ad-a720b344ab35","Type":"ContainerStarted","Data":"08fad185e03493d30dc17b9e2b71b635dfc321f1e693caf1bc1cb54f2736562f"} Nov 02 22:41:45.400918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:45.400870 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" podStartSLOduration=1.40085518 podStartE2EDuration="1.40085518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:41:45.399739232 +0000 UTC m=+4052.561834352" watchObservedRunningTime="2026-11-02 22:41:45.40085518 +0000 UTC m=+4052.562950348" Nov 02 22:41:46.294438 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:46.294401 5091 generic.go:356] "Generic (PLEG): container finished" podID="79f2d9f3-0a30-4857-83ad-a720b344ab35" containerID="f405db3bcece2ac2fd7bc1ed86edb277ab6db81f6387ddce0e6b75eed42a5c20" exitCode=0 Nov 02 22:41:46.294857 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:46.294485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" event={"ID":"79f2d9f3-0a30-4857-83ad-a720b344ab35","Type":"ContainerDied","Data":"f405db3bcece2ac2fd7bc1ed86edb277ab6db81f6387ddce0e6b75eed42a5c20"} Nov 02 22:41:47.434163 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.434136 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:47.548882 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.548851 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca\") pod \"79f2d9f3-0a30-4857-83ad-a720b344ab35\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " Nov 02 22:41:47.549029 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.548957 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca\") pod \"79f2d9f3-0a30-4857-83ad-a720b344ab35\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " Nov 02 22:41:47.549091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.549031 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd\" (UniqueName: \"kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd\") pod \"79f2d9f3-0a30-4857-83ad-a720b344ab35\" (UID: \"79f2d9f3-0a30-4857-83ad-a720b344ab35\") " Nov 02 22:41:47.549270 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.549244 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca" pod "79f2d9f3-0a30-4857-83ad-a720b344ab35" (UID: "79f2d9f3-0a30-4857-83ad-a720b344ab35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:41:47.551427 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.551367 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca" pod "79f2d9f3-0a30-4857-83ad-a720b344ab35" (UID: "79f2d9f3-0a30-4857-83ad-a720b344ab35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:47.551568 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.551549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd" pod "79f2d9f3-0a30-4857-83ad-a720b344ab35" (UID: "79f2d9f3-0a30-4857-83ad-a720b344ab35"). InnerVolumeSpecName "kube-api-access-rs9gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:47.650065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.650024 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:41:47.650065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.650062 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9gd\" (UniqueName: \"kubernetes.io/projected/79f2d9f3-0a30-4857-83ad-a720b344ab35-kube-api-access-rs9gd\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:41:47.650065 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:47.650073 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f2d9f3-0a30-4857-83ad-a720b344ab35-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:41:48.305499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:48.305463 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" Nov 02 22:41:48.305499 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:48.305482 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg" event={"ID":"79f2d9f3-0a30-4857-83ad-a720b344ab35","Type":"ContainerDied","Data":"08fad185e03493d30dc17b9e2b71b635dfc321f1e693caf1bc1cb54f2736562f"} Nov 02 22:41:48.305812 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:41:48.305526 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08fad185e03493d30dc17b9e2b71b635dfc321f1e693caf1bc1cb54f2736562f" Nov 02 22:42:05.172698 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:05.172639 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg"] Nov 02 22:42:05.174314 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:05.174290 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0f09fe-8c2wg"] Nov 02 22:42:06.975334 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:06.975296 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f2d9f3-0a30-4857-83ad-a720b344ab35" path="/var/lib/kubelet/pods/79f2d9f3-0a30-4857-83ad-a720b344ab35/volumes" Nov 02 22:42:07.111987 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:42:07.111952 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:42:37.121254 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:42:37.121214 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:42:42.171091 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.171052 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff"] Nov 02 22:42:42.171511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.171453 5091 cpu_manager.go:414] "containerMap: removing container" podUID="79f2d9f3-0a30-4857-83ad-a720b344ab35" containerName="revoke-keys" Nov 02 22:42:42.171511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.171465 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f2d9f3-0a30-4857-83ad-a720b344ab35" containerName="revoke-keys" Nov 02 22:42:42.171591 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.171542 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f2d9f3-0a30-4857-83ad-a720b344ab35" containerName="revoke-keys" Nov 02 22:42:42.174456 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.174437 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.231489 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.231455 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff"] Nov 02 22:42:42.275424 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.275395 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.275593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.275436 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlthm\" (UniqueName: \"kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.275593 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.275458 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.278622 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.278598 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:42:42.278622 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.278611 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:42:42.278810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.278626 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:42:42.278810 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.278653 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zcnzx\"" Nov 02 22:42:42.377050 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.377014 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.377249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.377062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlthm\" (UniqueName: \"kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.377249 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.377100 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.377876 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.377851 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.379312 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.379292 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.433401 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.433338 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlthm\" (UniqueName: \"kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm\") pod \"maas-api-revoke-sub-f03629db-tlhff\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.487046 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.487020 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:42.860461 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:42.860435 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff"] Nov 02 22:42:42.862420 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:42:42.862393 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a6075a_f924_4f45_8aec_58dee4f082f3.slice/crio-c90327792aa94fd001f7541b35c6e5fab3b29ba6e3cf327ad47de14dc8051aa1 WatchSource:0}: Error finding container c90327792aa94fd001f7541b35c6e5fab3b29ba6e3cf327ad47de14dc8051aa1: Status 404 returned error can't find the container with id c90327792aa94fd001f7541b35c6e5fab3b29ba6e3cf327ad47de14dc8051aa1 Nov 02 22:42:43.590347 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:43.590259 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" event={"ID":"78a6075a-f924-4f45-8aec-58dee4f082f3","Type":"ContainerStarted","Data":"993e57e92449a299f429d51c54b78e19c490417f3b8e9d68cd0aaf32c217c986"} Nov 02 22:42:43.590347 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:43.590313 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" event={"ID":"78a6075a-f924-4f45-8aec-58dee4f082f3","Type":"ContainerStarted","Data":"c90327792aa94fd001f7541b35c6e5fab3b29ba6e3cf327ad47de14dc8051aa1"} Nov 02 22:42:43.704468 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:43.704417 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" podStartSLOduration=1.7044033010000001 podStartE2EDuration="1.704403301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:42:43.702210229 +0000 UTC m=+4110.864305346" watchObservedRunningTime="2026-11-02 22:42:43.704403301 +0000 UTC m=+4110.866498455" Nov 02 22:42:44.596124 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:44.596091 5091 generic.go:356] "Generic (PLEG): container finished" podID="78a6075a-f924-4f45-8aec-58dee4f082f3" containerID="993e57e92449a299f429d51c54b78e19c490417f3b8e9d68cd0aaf32c217c986" exitCode=0 Nov 02 22:42:44.596507 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:44.596146 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" event={"ID":"78a6075a-f924-4f45-8aec-58dee4f082f3","Type":"ContainerDied","Data":"993e57e92449a299f429d51c54b78e19c490417f3b8e9d68cd0aaf32c217c986"} Nov 02 22:42:45.735928 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.735905 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:45.810511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.810484 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm\" (UniqueName: \"kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm\") pod \"78a6075a-f924-4f45-8aec-58dee4f082f3\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " Nov 02 22:42:45.810658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.810530 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca\") pod \"78a6075a-f924-4f45-8aec-58dee4f082f3\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " Nov 02 22:42:45.810658 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.810601 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca\") pod \"78a6075a-f924-4f45-8aec-58dee4f082f3\" (UID: \"78a6075a-f924-4f45-8aec-58dee4f082f3\") " Nov 02 22:42:45.810970 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.810945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca" pod "78a6075a-f924-4f45-8aec-58dee4f082f3" (UID: "78a6075a-f924-4f45-8aec-58dee4f082f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:45.812515 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.812484 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm" pod "78a6075a-f924-4f45-8aec-58dee4f082f3" (UID: "78a6075a-f924-4f45-8aec-58dee4f082f3"). InnerVolumeSpecName "kube-api-access-jlthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:45.812606 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.812516 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca" pod "78a6075a-f924-4f45-8aec-58dee4f082f3" (UID: "78a6075a-f924-4f45-8aec-58dee4f082f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:42:45.912282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.912252 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlthm\" (UniqueName: \"kubernetes.io/projected/78a6075a-f924-4f45-8aec-58dee4f082f3-kube-api-access-jlthm\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:45.912282 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.912280 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:45.912492 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:45.912290 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a6075a-f924-4f45-8aec-58dee4f082f3-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:46.607578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:46.607542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" event={"ID":"78a6075a-f924-4f45-8aec-58dee4f082f3","Type":"ContainerDied","Data":"c90327792aa94fd001f7541b35c6e5fab3b29ba6e3cf327ad47de14dc8051aa1"} Nov 02 22:42:46.607578 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:46.607562 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tlhff" Nov 02 22:42:46.607822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:42:46.607591 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90327792aa94fd001f7541b35c6e5fab3b29ba6e3cf327ad47de14dc8051aa1" Nov 02 22:43:02.579919 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.579881 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs"] Nov 02 22:43:02.580556 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.580534 5091 cpu_manager.go:414] "containerMap: removing container" podUID="78a6075a-f924-4f45-8aec-58dee4f082f3" containerName="revoke-keys" Nov 02 22:43:02.580604 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.580560 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a6075a-f924-4f45-8aec-58dee4f082f3" containerName="revoke-keys" Nov 02 22:43:02.580731 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.580719 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a6075a-f924-4f45-8aec-58dee4f082f3" containerName="revoke-keys" Nov 02 22:43:02.584434 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.584410 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.637609 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.637533 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:43:02.687168 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.687137 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:43:02.689365 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.689345 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs"] Nov 02 22:43:02.698149 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.698123 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:43:02.698256 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.698185 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zcnzx\"" Nov 02 22:43:02.706077 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.706054 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2sx7\" (UniqueName: \"kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.706176 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.706109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.706254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.706229 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.807337 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.807311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.807511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.807356 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.807511 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.807425 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2sx7\" (UniqueName: \"kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.808125 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.808098 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.809882 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.809858 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.863837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.863781 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2sx7\" (UniqueName: \"kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7\") pod \"maas-api-revoke-sub-0165d7d2-mqtxs\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:02.897638 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:02.897615 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:03.266747 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:03.266722 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs"] Nov 02 22:43:03.269324 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:43:03.269295 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2897fa77_42a8_4fc6_8bbd_efba80506219.slice/crio-a2bd1983e8078cbc90b6d79e4587d2d71e624aaab4aba2ddef53897dbf682c8a WatchSource:0}: Error finding container a2bd1983e8078cbc90b6d79e4587d2d71e624aaab4aba2ddef53897dbf682c8a: Status 404 returned error can't find the container with id a2bd1983e8078cbc90b6d79e4587d2d71e624aaab4aba2ddef53897dbf682c8a Nov 02 22:43:03.741595 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:03.741558 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" event={"ID":"2897fa77-42a8-4fc6-8bbd-efba80506219","Type":"ContainerStarted","Data":"7d1ad0421f3fa7e904bb5bb93f948fc069d557dd657f30bae40fc897a5eda151"} Nov 02 22:43:03.741993 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:03.741612 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" event={"ID":"2897fa77-42a8-4fc6-8bbd-efba80506219","Type":"ContainerStarted","Data":"a2bd1983e8078cbc90b6d79e4587d2d71e624aaab4aba2ddef53897dbf682c8a"} Nov 02 22:43:03.856296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:03.856205 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" podStartSLOduration=1.8561904930000002 podStartE2EDuration="1.856190493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:03.855833039 +0000 UTC m=+4131.017928160" watchObservedRunningTime="2026-11-02 22:43:03.856190493 +0000 UTC m=+4131.018285610" Nov 02 22:43:04.747998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:04.747960 5091 generic.go:356] "Generic (PLEG): container finished" podID="2897fa77-42a8-4fc6-8bbd-efba80506219" containerID="7d1ad0421f3fa7e904bb5bb93f948fc069d557dd657f30bae40fc897a5eda151" exitCode=0 Nov 02 22:43:04.748392 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:04.748046 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" event={"ID":"2897fa77-42a8-4fc6-8bbd-efba80506219","Type":"ContainerDied","Data":"7d1ad0421f3fa7e904bb5bb93f948fc069d557dd657f30bae40fc897a5eda151"} Nov 02 22:43:05.884997 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:05.884972 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:06.036358 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.036281 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca\") pod \"2897fa77-42a8-4fc6-8bbd-efba80506219\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " Nov 02 22:43:06.036521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.036395 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca\") pod \"2897fa77-42a8-4fc6-8bbd-efba80506219\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " Nov 02 22:43:06.036521 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.036469 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7\" (UniqueName: \"kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7\") pod \"2897fa77-42a8-4fc6-8bbd-efba80506219\" (UID: \"2897fa77-42a8-4fc6-8bbd-efba80506219\") " Nov 02 22:43:06.036723 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.036693 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca" pod "2897fa77-42a8-4fc6-8bbd-efba80506219" (UID: "2897fa77-42a8-4fc6-8bbd-efba80506219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:43:06.038457 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.038436 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca" pod "2897fa77-42a8-4fc6-8bbd-efba80506219" (UID: "2897fa77-42a8-4fc6-8bbd-efba80506219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:43:06.038583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.038564 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7" pod "2897fa77-42a8-4fc6-8bbd-efba80506219" (UID: "2897fa77-42a8-4fc6-8bbd-efba80506219"). InnerVolumeSpecName "kube-api-access-c2sx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:43:06.137763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.137732 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:43:06.137763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.137755 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2sx7\" (UniqueName: \"kubernetes.io/projected/2897fa77-42a8-4fc6-8bbd-efba80506219-kube-api-access-c2sx7\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:43:06.137763 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.137766 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2897fa77-42a8-4fc6-8bbd-efba80506219-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:43:06.759025 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.758988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" event={"ID":"2897fa77-42a8-4fc6-8bbd-efba80506219","Type":"ContainerDied","Data":"a2bd1983e8078cbc90b6d79e4587d2d71e624aaab4aba2ddef53897dbf682c8a"} Nov 02 22:43:06.759025 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.759012 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mqtxs" Nov 02 22:43:06.759251 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:06.759044 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2bd1983e8078cbc90b6d79e4587d2d71e624aaab4aba2ddef53897dbf682c8a" Nov 02 22:43:07.111378 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:43:07.111296 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:43:14.252248 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.252198 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn"] Nov 02 22:43:14.252922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.252639 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2897fa77-42a8-4fc6-8bbd-efba80506219" containerName="revoke-keys" Nov 02 22:43:14.252922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.252651 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2897fa77-42a8-4fc6-8bbd-efba80506219" containerName="revoke-keys" Nov 02 22:43:14.252922 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.252745 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2897fa77-42a8-4fc6-8bbd-efba80506219" containerName="revoke-keys" Nov 02 22:43:14.256087 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.256065 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.359735 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.359703 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:43:14.359924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.359835 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zcnzx\"" Nov 02 22:43:14.359924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.359849 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:43:14.359924 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.359900 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:43:14.361927 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.361904 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn"] Nov 02 22:43:14.409460 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.409427 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.409631 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.409542 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.409631 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.409576 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxz6\" (UniqueName: \"kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.510263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.510183 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.510263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.510229 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.510263 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.510250 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxz6\" (UniqueName: \"kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.510987 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.510966 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.512819 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.512791 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.567032 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.566998 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxz6\" (UniqueName: \"kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6\") pod \"maas-api-revoke-sub-6b19338f-cvxnn\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.568781 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.568761 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:14.938954 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:14.938930 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn"] Nov 02 22:43:14.940929 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:43:14.940899 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e71f0a_5577_437c_8a0f_314ee30d7027.slice/crio-d82bf4ca71e8e6ef2d40809ad74fb563becac806351e4a241d336bf9491d02bf WatchSource:0}: Error finding container d82bf4ca71e8e6ef2d40809ad74fb563becac806351e4a241d336bf9491d02bf: Status 404 returned error can't find the container with id d82bf4ca71e8e6ef2d40809ad74fb563becac806351e4a241d336bf9491d02bf Nov 02 22:43:15.804278 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:15.804237 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" event={"ID":"05e71f0a-5577-437c-8a0f-314ee30d7027","Type":"ContainerStarted","Data":"44cc4715a04c61dc5c2bb976627e78615be3ad699763d2fca5645ac720843e59"} Nov 02 22:43:15.804708 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:15.804299 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" event={"ID":"05e71f0a-5577-437c-8a0f-314ee30d7027","Type":"ContainerStarted","Data":"d82bf4ca71e8e6ef2d40809ad74fb563becac806351e4a241d336bf9491d02bf"} Nov 02 22:43:15.918605 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:15.918555 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" podStartSLOduration=1.918540227 podStartE2EDuration="1.918540227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:15.917056266 +0000 UTC m=+4143.079151395" watchObservedRunningTime="2026-11-02 22:43:15.918540227 +0000 UTC m=+4143.080635348" Nov 02 22:43:16.810845 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:16.810808 5091 generic.go:356] "Generic (PLEG): container finished" podID="05e71f0a-5577-437c-8a0f-314ee30d7027" containerID="44cc4715a04c61dc5c2bb976627e78615be3ad699763d2fca5645ac720843e59" exitCode=0 Nov 02 22:43:16.811254 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:16.810888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" event={"ID":"05e71f0a-5577-437c-8a0f-314ee30d7027","Type":"ContainerDied","Data":"44cc4715a04c61dc5c2bb976627e78615be3ad699763d2fca5645ac720843e59"} Nov 02 22:43:17.952344 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:17.952316 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:18.147709 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.147659 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca\") pod \"05e71f0a-5577-437c-8a0f-314ee30d7027\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " Nov 02 22:43:18.147900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.147751 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca\") pod \"05e71f0a-5577-437c-8a0f-314ee30d7027\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " Nov 02 22:43:18.147900 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.147800 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6\" (UniqueName: \"kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6\") pod \"05e71f0a-5577-437c-8a0f-314ee30d7027\" (UID: \"05e71f0a-5577-437c-8a0f-314ee30d7027\") " Nov 02 22:43:18.148154 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.148130 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca" pod "05e71f0a-5577-437c-8a0f-314ee30d7027" (UID: "05e71f0a-5577-437c-8a0f-314ee30d7027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:43:18.149837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.149814 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca" pod "05e71f0a-5577-437c-8a0f-314ee30d7027" (UID: "05e71f0a-5577-437c-8a0f-314ee30d7027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:43:18.150129 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.149897 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6" pod "05e71f0a-5577-437c-8a0f-314ee30d7027" (UID: "05e71f0a-5577-437c-8a0f-314ee30d7027"). InnerVolumeSpecName "kube-api-access-mbxz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:43:18.249074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.249050 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-xks-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:43:18.249074 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.249073 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e71f0a-5577-437c-8a0f-314ee30d7027-maas-api-openshift-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:43:18.249221 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.249083 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxz6\" (UniqueName: \"kubernetes.io/projected/05e71f0a-5577-437c-8a0f-314ee30d7027-kube-api-access-mbxz6\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:43:18.822371 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.822335 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" Nov 02 22:43:18.822558 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.822366 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvxnn" event={"ID":"05e71f0a-5577-437c-8a0f-314ee30d7027","Type":"ContainerDied","Data":"d82bf4ca71e8e6ef2d40809ad74fb563becac806351e4a241d336bf9491d02bf"} Nov 02 22:43:18.822558 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:18.822415 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d82bf4ca71e8e6ef2d40809ad74fb563becac806351e4a241d336bf9491d02bf" Nov 02 22:43:26.503166 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:26.503130 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-555867774b-47zn5_d8649ce6-c528-455c-81b4-183e5012d436/authorino/0.log" Nov 02 22:43:28.872847 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:28.872808 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m2sxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:43:28.873489 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:28.873446 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m2sxm" podUID="ebbdf6bd-a942-433c-a08e-ca09206efe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:43:29.103911 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:29.103872 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xbnfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:43:29.104102 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:29.103954 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xbnfm" podUID="5b93f41f-2956-4052-a31c-ae1aded9a235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:43:37.111749 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:43:37.111712 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:43:37.899475 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:37.899445 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vggpx_a68a0a17-418b-4163-9d20-42f126791bb8/manager/0.log" Nov 02 22:43:39.340356 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:39.340328 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ss7kj_2a821de6-21e0-4677-bb47-5e2863b4d4c5/proxy/0.log" Nov 02 22:43:40.754506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:40.754477 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-mqtxs_2897fa77-42a8-4fc6-8bbd-efba80506219/revoke-keys/0.log" Nov 02 22:43:40.960787 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:40.960760 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cvxnn_05e71f0a-5577-437c-8a0f-314ee30d7027/revoke-keys/0.log" Nov 02 22:43:41.177303 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:41.177278 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tlhff_78a6075a-f924-4f45-8aec-58dee4f082f3/revoke-keys/0.log" Nov 02 22:43:42.785175 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:42.785145 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-555867774b-47zn5_d8649ce6-c528-455c-81b4-183e5012d436/authorino/0.log" Nov 02 22:43:43.245378 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:43.245348 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nc4jw_d31b49e2-0bd4-4d60-9094-92628656dfec/manager/0.log" Nov 02 22:43:43.463836 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:43.463805 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cn7qk_186424fc-e8af-415d-8ed6-501e3640f01d/manager/0.log" Nov 02 22:43:44.881870 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:44.881842 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ns97n_3244c502-b53c-4193-9e0d-3507245016ca/manager/0.log" Nov 02 22:43:45.679504 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:45.679477 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd_769f1cb9-13f8-43f9-9940-8ba271e9181f/istio-proxy/0.log" Nov 02 22:43:47.158306 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:47.158277 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-m2sxm_ebbdf6bd-a942-433c-a08e-ca09206efe34/istio-proxy/0.log" Nov 02 22:43:48.683908 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:48.683874 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xbnfm_5b93f41f-2956-4052-a31c-ae1aded9a235/istio-proxy/0.log" Nov 02 22:43:48.985373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:48.985293 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-zw2ss_b7a3bf38-6dc8-4465-83ba-9eeebe01a791/istio-proxy/0.log" Nov 02 22:43:49.815023 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:49.814992 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd7d6d4d5-p5mwz_24c005d3-f7c3-4292-ac2d-ff26ec4065fd/router/0.log" Nov 02 22:43:53.231286 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:43:53.231252 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-555867774b-47zn5_d8649ce6-c528-455c-81b4-183e5012d436/authorino/0.log" Nov 02 22:44:02.821999 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:02.821908 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vggpx_a68a0a17-418b-4163-9d20-42f126791bb8/manager/0.log" Nov 02 22:44:04.041923 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:04.041887 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ss7kj_2a821de6-21e0-4677-bb47-5e2863b4d4c5/proxy/0.log" Nov 02 22:44:05.128519 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:05.128488 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-mqtxs_2897fa77-42a8-4fc6-8bbd-efba80506219/revoke-keys/0.log" Nov 02 22:44:05.349583 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:05.349553 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cvxnn_05e71f0a-5577-437c-8a0f-314ee30d7027/revoke-keys/0.log" Nov 02 22:44:05.553178 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:05.553098 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tlhff_78a6075a-f924-4f45-8aec-58dee4f082f3/revoke-keys/0.log" Nov 02 22:44:06.840030 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:06.840001 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-555867774b-47zn5_d8649ce6-c528-455c-81b4-183e5012d436/authorino/0.log" Nov 02 22:44:07.112546 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:44:07.112463 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:44:07.289276 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:07.289250 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nc4jw_d31b49e2-0bd4-4d60-9094-92628656dfec/manager/0.log" Nov 02 22:44:07.507448 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:07.507423 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cn7qk_186424fc-e8af-415d-8ed6-501e3640f01d/manager/0.log" Nov 02 22:44:08.607830 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:08.607799 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ns97n_3244c502-b53c-4193-9e0d-3507245016ca/manager/0.log" Nov 02 22:44:09.254075 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:09.254047 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd_769f1cb9-13f8-43f9-9940-8ba271e9181f/istio-proxy/0.log" Nov 02 22:44:10.721838 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:10.721808 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-m2sxm_ebbdf6bd-a942-433c-a08e-ca09206efe34/istio-proxy/0.log" Nov 02 22:44:12.260139 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:12.260105 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xbnfm_5b93f41f-2956-4052-a31c-ae1aded9a235/istio-proxy/0.log" Nov 02 22:44:12.580225 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:12.580147 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-zw2ss_b7a3bf38-6dc8-4465-83ba-9eeebe01a791/istio-proxy/0.log" Nov 02 22:44:13.474616 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:13.474588 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cd7d6d4d5-p5mwz_24c005d3-f7c3-4292-ac2d-ff26ec4065fd/router/0.log" Nov 02 22:44:13.686002 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:13.685971 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:44:13.699738 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:44:13.699711 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kklpk_82eb2931-ff21-4bf4-9589-995055842cb4/console-operator/3.log" Nov 02 22:44:37.111948 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:44:37.111908 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:45:00.168345 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.168311 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26"] Nov 02 22:45:00.168972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.168726 5091 cpu_manager.go:414] "containerMap: removing container" podUID="05e71f0a-5577-437c-8a0f-314ee30d7027" containerName="revoke-keys" Nov 02 22:45:00.168972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.168739 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e71f0a-5577-437c-8a0f-314ee30d7027" containerName="revoke-keys" Nov 02 22:45:00.168972 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.168823 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e71f0a-5577-437c-8a0f-314ee30d7027" containerName="revoke-keys" Nov 02 22:45:00.171884 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.171867 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.223875 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.223851 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:45:00.273462 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.273436 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zcnzx\"" Nov 02 22:45:00.273611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.273483 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:45:00.275652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.275631 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26"] Nov 02 22:45:00.335181 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.335154 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczn7\" (UniqueName: \"kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7\") pod \"maas-api-key-cleanup-29894325-5rz26\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.335307 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.335230 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894325-5rz26\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.435935 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.435876 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894325-5rz26\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.435935 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.435927 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczn7\" (UniqueName: \"kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7\") pod \"maas-api-key-cleanup-29894325-5rz26\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.436509 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.436492 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894325-5rz26\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.492714 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.492659 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczn7\" (UniqueName: \"kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7\") pod \"maas-api-key-cleanup-29894325-5rz26\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:00.784296 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:00.784212 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:01.155328 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:01.155304 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26"] Nov 02 22:45:01.156957 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:45:01.156931 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad73cb4b_b7f1_467c_b33d_77244395f213.slice/crio-c37440337c17c0db983749d7867ed47051fc58b92c3c880c19be38c20c4a2da2 WatchSource:0}: Error finding container c37440337c17c0db983749d7867ed47051fc58b92c3c880c19be38c20c4a2da2: Status 404 returned error can't find the container with id c37440337c17c0db983749d7867ed47051fc58b92c3c880c19be38c20c4a2da2 Nov 02 22:45:01.387975 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:01.387943 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" event={"ID":"ad73cb4b-b7f1-467c-b33d-77244395f213","Type":"ContainerStarted","Data":"c37440337c17c0db983749d7867ed47051fc58b92c3c880c19be38c20c4a2da2"} Nov 02 22:45:02.395694 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:02.395651 5091 generic.go:356] "Generic (PLEG): container finished" podID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerID="0dcdd430afceb5efed495968fc7c1549a5b35f4a2893d3cdde4a95928641cf06" exitCode=60 Nov 02 22:45:02.396113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:02.395739 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" event={"ID":"ad73cb4b-b7f1-467c-b33d-77244395f213","Type":"ContainerDied","Data":"0dcdd430afceb5efed495968fc7c1549a5b35f4a2893d3cdde4a95928641cf06"} Nov 02 22:45:02.396113 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:02.396001 5091 scope.go:122] "RemoveContainer" containerID="0dcdd430afceb5efed495968fc7c1549a5b35f4a2893d3cdde4a95928641cf06" Nov 02 22:45:03.402852 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:03.402819 5091 generic.go:356] "Generic (PLEG): container finished" podID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerID="d0dc77cf020a61be812e25bd942cb069681bcd2080a12803def8f1a7e48a767a" exitCode=60 Nov 02 22:45:03.403289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:03.402902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" event={"ID":"ad73cb4b-b7f1-467c-b33d-77244395f213","Type":"ContainerDied","Data":"d0dc77cf020a61be812e25bd942cb069681bcd2080a12803def8f1a7e48a767a"} Nov 02 22:45:03.403289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:03.402956 5091 scope.go:122] "RemoveContainer" containerID="0dcdd430afceb5efed495968fc7c1549a5b35f4a2893d3cdde4a95928641cf06" Nov 02 22:45:03.403289 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:03.403183 5091 scope.go:122] "RemoveContainer" containerID="d0dc77cf020a61be812e25bd942cb069681bcd2080a12803def8f1a7e48a767a" Nov 02 22:45:03.403443 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:45:03.403422 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894325-5rz26_odh-ai-gateway-infra(ad73cb4b-b7f1-467c-b33d-77244395f213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" Nov 02 22:45:04.409020 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:04.408993 5091 scope.go:122] "RemoveContainer" containerID="d0dc77cf020a61be812e25bd942cb069681bcd2080a12803def8f1a7e48a767a" Nov 02 22:45:04.409437 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:45:04.409173 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894325-5rz26_odh-ai-gateway-infra(ad73cb4b-b7f1-467c-b33d-77244395f213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" Nov 02 22:45:07.111950 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:45:07.111911 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:45:16.975116 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:16.975085 5091 scope.go:122] "RemoveContainer" containerID="d0dc77cf020a61be812e25bd942cb069681bcd2080a12803def8f1a7e48a767a" Nov 02 22:45:17.475285 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:17.475250 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" event={"ID":"ad73cb4b-b7f1-467c-b33d-77244395f213","Type":"ContainerStarted","Data":"70b053f3db50cb437d8b5b2cc605fb8e46ccd9e4fb7baf620cfbd0eb9765108c"} Nov 02 22:45:17.589855 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:17.589761 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" podStartSLOduration=17.589747621 podStartE2EDuration="17.589747621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:17.587002862 +0000 UTC m=+4264.749097981" watchObservedRunningTime="2026-11-02 22:45:17.589747621 +0000 UTC m=+4264.751842741" Nov 02 22:45:18.482389 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:18.482356 5091 generic.go:356] "Generic (PLEG): container finished" podID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerID="70b053f3db50cb437d8b5b2cc605fb8e46ccd9e4fb7baf620cfbd0eb9765108c" exitCode=60 Nov 02 22:45:18.482892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:18.482426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" event={"ID":"ad73cb4b-b7f1-467c-b33d-77244395f213","Type":"ContainerDied","Data":"70b053f3db50cb437d8b5b2cc605fb8e46ccd9e4fb7baf620cfbd0eb9765108c"} Nov 02 22:45:18.482892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:18.482482 5091 scope.go:122] "RemoveContainer" containerID="d0dc77cf020a61be812e25bd942cb069681bcd2080a12803def8f1a7e48a767a" Nov 02 22:45:18.482892 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:18.482841 5091 scope.go:122] "RemoveContainer" containerID="70b053f3db50cb437d8b5b2cc605fb8e46ccd9e4fb7baf620cfbd0eb9765108c" Nov 02 22:45:18.483080 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:45:18.483061 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894325-5rz26_odh-ai-gateway-infra(ad73cb4b-b7f1-467c-b33d-77244395f213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" Nov 02 22:45:18.597207 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:18.597180 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26"] Nov 02 22:45:19.624111 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.624089 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:19.694692 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.694647 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca\") pod \"ad73cb4b-b7f1-467c-b33d-77244395f213\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " Nov 02 22:45:19.694820 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.694768 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7\" (UniqueName: \"kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7\") pod \"ad73cb4b-b7f1-467c-b33d-77244395f213\" (UID: \"ad73cb4b-b7f1-467c-b33d-77244395f213\") " Nov 02 22:45:19.694960 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.694939 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca" pod "ad73cb4b-b7f1-467c-b33d-77244395f213" (UID: "ad73cb4b-b7f1-467c-b33d-77244395f213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:19.696785 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.696762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7" pod "ad73cb4b-b7f1-467c-b33d-77244395f213" (UID: "ad73cb4b-b7f1-467c-b33d-77244395f213"). InnerVolumeSpecName "kube-api-access-xczn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:19.795664 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.795588 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xczn7\" (UniqueName: \"kubernetes.io/projected/ad73cb4b-b7f1-467c-b33d-77244395f213-kube-api-access-xczn7\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:45:19.795664 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:19.795617 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad73cb4b-b7f1-467c-b33d-77244395f213-maas-api-service-ca\") on node \"ip-10-0-128-36.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:45:20.496110 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:20.496079 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" Nov 02 22:45:20.496301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:20.496115 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26" event={"ID":"ad73cb4b-b7f1-467c-b33d-77244395f213","Type":"ContainerDied","Data":"c37440337c17c0db983749d7867ed47051fc58b92c3c880c19be38c20c4a2da2"} Nov 02 22:45:20.496301 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:20.496171 5091 scope.go:122] "RemoveContainer" containerID="70b053f3db50cb437d8b5b2cc605fb8e46ccd9e4fb7baf620cfbd0eb9765108c" Nov 02 22:45:20.659750 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:20.659718 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26"] Nov 02 22:45:20.709245 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:20.709214 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-5rz26"] Nov 02 22:45:20.975858 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:20.975826 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" path="/var/lib/kubelet/pods/ad73cb4b-b7f1-467c-b33d-77244395f213/volumes" Nov 02 22:45:25.885424 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.885346 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lxt6h/must-gather-4b7xf"] Nov 02 22:45:25.885883 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.885865 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.885929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.885886 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.885929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.885909 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.885929 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.885915 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.886034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.886014 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.886034 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.886025 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.886108 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.886098 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.886158 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.886107 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.886192 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.886182 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad73cb4b-b7f1-467c-b33d-77244395f213" containerName="cleanup" Nov 02 22:45:25.890452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.890430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:25.936410 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.936389 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e-must-gather-output\") pod \"must-gather-4b7xf\" (UID: \"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e\") " pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:25.936517 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.936440 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnmf\" (UniqueName: \"kubernetes.io/projected/615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e-kube-api-access-rnnmf\") pod \"must-gather-4b7xf\" (UID: \"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e\") " pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:25.946409 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.946384 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lxt6h/must-gather-4b7xf"] Nov 02 22:45:25.993373 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.993348 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lxt6h\"/\"kube-root-ca.crt\"" Nov 02 22:45:25.993502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.993421 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lxt6h\"/\"default-dockercfg-q6zgg\"" Nov 02 22:45:25.993502 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:25.993468 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lxt6h\"/\"openshift-service-ca.crt\"" Nov 02 22:45:26.037506 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.037481 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnmf\" (UniqueName: \"kubernetes.io/projected/615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e-kube-api-access-rnnmf\") pod \"must-gather-4b7xf\" (UID: \"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e\") " pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:26.037643 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.037586 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e-must-gather-output\") pod \"must-gather-4b7xf\" (UID: \"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e\") " pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:26.037937 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.037917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e-must-gather-output\") pod \"must-gather-4b7xf\" (UID: \"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e\") " pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:26.093584 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.093559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnmf\" (UniqueName: \"kubernetes.io/projected/615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e-kube-api-access-rnnmf\") pod \"must-gather-4b7xf\" (UID: \"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e\") " pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:26.204134 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.204062 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lxt6h/must-gather-4b7xf" Nov 02 22:45:26.577467 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.577443 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lxt6h/must-gather-4b7xf"] Nov 02 22:45:26.579638 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:45:26.579603 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod615b90aa_b7e2_4cdb_82a2_eb385dfd2b9e.slice/crio-b3379f82463c07ade28b862677ea737ae4d7000981a5645a7fa2483d2755e6d3 WatchSource:0}: Error finding container b3379f82463c07ade28b862677ea737ae4d7000981a5645a7fa2483d2755e6d3: Status 404 returned error can't find the container with id b3379f82463c07ade28b862677ea737ae4d7000981a5645a7fa2483d2755e6d3 Nov 02 22:45:26.581465 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:26.581447 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:45:27.534089 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:27.534052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/must-gather-4b7xf" event={"ID":"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e","Type":"ContainerStarted","Data":"b3379f82463c07ade28b862677ea737ae4d7000981a5645a7fa2483d2755e6d3"} Nov 02 22:45:28.540283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:28.540248 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/must-gather-4b7xf" event={"ID":"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e","Type":"ContainerStarted","Data":"ef52e51824c2d18eb532d5e8718e0e6860fa89517f1c9945a1131fd3fed81fea"} Nov 02 22:45:28.540283 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:28.540292 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/must-gather-4b7xf" event={"ID":"615b90aa-b7e2-4cdb-82a2-eb385dfd2b9e","Type":"ContainerStarted","Data":"ec5d5e19d0fb1aafe1a12f858a08d119a8bb88b5602667e7659f537e25ad1258"} Nov 02 22:45:28.652575 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:28.652523 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lxt6h/must-gather-4b7xf" podStartSLOduration=2.465151598 podStartE2EDuration="3.652507734s" totalImagesPullingTime="1.187356136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:28.651936564 +0000 UTC m=+4275.814031686" watchObservedRunningTime="2026-11-02 22:45:28.652507734 +0000 UTC m=+4275.814602851" Nov 02 22:45:36.058837 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:36.058799 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sr2gb_4016e9e3-e3e8-454a-a635-49970015c350/global-pull-secret-syncer/0.log" Nov 02 22:45:36.655724 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:36.648258 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lzqpt_31e0fa56-e920-4006-91c5-7425cd741ead/connection-checker/0.log" Nov 02 22:45:37.079740 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:37.078943 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nkwv9_7d7f8a45-6604-4c5b-9b1d-e761aa0ce16b/connection-checker/0.log" Nov 02 22:45:37.177702 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:45:37.174867 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:45:37.509756 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:37.508429 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xxjw8_06076e5c-0467-47fa-a47b-946e170df5bb/connection-checker/0.log" Nov 02 22:45:38.991642 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:38.991601 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wk8sg_eb533acd-73cb-4f30-a60e-e96debb69f25/konnectivity-agent/0.log" Nov 02 22:45:39.390902 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:39.390870 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-36.us-west-1.compute.internal_356429dd5260af9a132a468067041120/haproxy/0.log" Nov 02 22:45:44.265581 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.265526 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-555867774b-47zn5_d8649ce6-c528-455c-81b4-183e5012d436/authorino/0.log" Nov 02 22:45:44.582697 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.582609 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nc4jw_d31b49e2-0bd4-4d60-9094-92628656dfec/manager/0.log" Nov 02 22:45:44.852582 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.852506 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp"] Nov 02 22:45:44.859611 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.859581 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:44.906930 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.906891 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd59s\" (UniqueName: \"kubernetes.io/projected/7a607dfc-4855-4a8a-843a-004579285ac3-kube-api-access-jd59s\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:44.907083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.906950 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-podres\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:44.907083 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.906979 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-sys\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:44.907171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.907109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-lib-modules\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:44.907171 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.907142 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-proc\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:44.912918 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.912886 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp"] Nov 02 22:45:44.997498 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:44.997442 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cn7qk_186424fc-e8af-415d-8ed6-501e3640f01d/manager/0.log" Nov 02 22:45:45.008424 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008398 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-sys\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-lib-modules\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008463 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-proc\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008531 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008521 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd59s\" (UniqueName: \"kubernetes.io/projected/7a607dfc-4855-4a8a-843a-004579285ac3-kube-api-access-jd59s\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008524 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-sys\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-lib-modules\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008605 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-podres\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008652 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008613 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-proc\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.008822 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.008704 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a607dfc-4855-4a8a-843a-004579285ac3-podres\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.064998 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.064967 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd59s\" (UniqueName: \"kubernetes.io/projected/7a607dfc-4855-4a8a-843a-004579285ac3-kube-api-access-jd59s\") pod \"perf-node-gather-daemonset-4sxkp\" (UID: \"7a607dfc-4855-4a8a-843a-004579285ac3\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.175648 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.175614 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.549370 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.549282 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp"] Nov 02 22:45:45.553696 ip-10-0-128-36 kubenswrapper[5091]: W1102 22:45:45.553635 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a607dfc_4855_4a8a_843a_004579285ac3.slice/crio-7e60f127a8d1acb1c726a64ea15c73a286bfb53826051cceb9dea471029871c6 WatchSource:0}: Error finding container 7e60f127a8d1acb1c726a64ea15c73a286bfb53826051cceb9dea471029871c6: Status 404 returned error can't find the container with id 7e60f127a8d1acb1c726a64ea15c73a286bfb53826051cceb9dea471029871c6 Nov 02 22:45:45.731459 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.731429 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" event={"ID":"7a607dfc-4855-4a8a-843a-004579285ac3","Type":"ContainerStarted","Data":"c3f9fa57e9594b667f0d68ba73c41be1e5eed44829092650253fce8f9a6125a6"} Nov 02 22:45:45.731573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.731481 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" event={"ID":"7a607dfc-4855-4a8a-843a-004579285ac3","Type":"ContainerStarted","Data":"7e60f127a8d1acb1c726a64ea15c73a286bfb53826051cceb9dea471029871c6"} Nov 02 22:45:45.731573 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.731563 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:45.844574 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:45.844279 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" podStartSLOduration=1.844261376 podStartE2EDuration="1.844261376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:45.843206385 +0000 UTC m=+4293.005301506" watchObservedRunningTime="2026-11-02 22:45:45.844261376 +0000 UTC m=+4293.006356496" Nov 02 22:45:47.003452 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:47.003415 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ns97n_3244c502-b53c-4193-9e0d-3507245016ca/manager/0.log" Nov 02 22:45:51.749007 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:51.748976 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-4sxkp" Nov 02 22:45:52.519849 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:52.519816 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-rrvdw_0f0291ae-a1d6-4312-9e07-da45a493466e/cluster-monitoring-operator/0.log" Nov 02 22:45:55.047084 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:55.047047 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhzhl_8be30588-9291-4133-82a4-3a6df47e2b4a/node-exporter/0.log" Nov 02 22:45:55.458933 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:55.458904 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhzhl_8be30588-9291-4133-82a4-3a6df47e2b4a/kube-rbac-proxy/0.log" Nov 02 22:45:55.875096 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:45:55.875062 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhzhl_8be30588-9291-4133-82a4-3a6df47e2b4a/init-textfile/0.log" Nov 02 22:46:07.120349 ip-10-0-128-36 kubenswrapper[5091]: E1102 22:46:07.120310 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b5tqm" containerName="extractor" Nov 02 22:46:07.277292 ip-10-0-128-36 kubenswrapper[5091]: I1102 22:46:07.277263 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-k4vhd_769f1cb9-13f8-43f9-9940-8ba271e9181f/istio-proxy/0.log"